2025-01-02 07:48:44,209 INFO StreamThr :2914664 [internal.py:wandb_internal():85] W&B internal server running at pid: 2914664, started at: 2025-01-02 07:48:44.208088 2025-01-02 07:48:44,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status 2025-01-02 07:48:44,218 INFO WriterThread:2914664 [datastore.py:open_for_write():87] open: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/run-1ecgrehs.wandb 2025-01-02 07:48:44,219 DEBUG SenderThread:2914664 [sender.py:send():379] send: header 2025-01-02 07:48:44,288 DEBUG SenderThread:2914664 [sender.py:send():379] send: run 2025-01-02 07:48:44,635 INFO SenderThread:2914664 [dir_watcher.py:__init__():211] watching files in: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files 2025-01-02 07:48:44,636 INFO SenderThread:2914664 [sender.py:_start_run_threads():1188] run started: 1ecgrehs with start time 1735804124.21017 2025-01-02 07:48:44,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: check_version 2025-01-02 07:48:44,661 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: check_version 2025-01-02 07:48:44,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: run_start 2025-01-02 07:48:44,776 DEBUG HandlerThread:2914664 [system_info.py:__init__():26] System info init 2025-01-02 07:48:44,776 DEBUG HandlerThread:2914664 [system_info.py:__init__():41] System info init done 2025-01-02 07:48:44,776 INFO HandlerThread:2914664 [system_monitor.py:start():194] Starting system monitor 2025-01-02 07:48:44,776 INFO SystemMonitor:2914664 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-02 07:48:44,777 INFO HandlerThread:2914664 [system_monitor.py:probe():214] Collecting system info 2025-01-02 07:48:44,777 INFO SystemMonitor:2914664 [interfaces.py:start():188] Started cpu monitoring 2025-01-02 07:48:44,778 INFO SystemMonitor:2914664 [interfaces.py:start():188] Started disk monitoring 2025-01-02 07:48:44,779 INFO SystemMonitor:2914664 [interfaces.py:start():188] Started gpu monitoring 2025-01-02 07:48:44,780 INFO SystemMonitor:2914664 [interfaces.py:start():188] Started memory monitoring 2025-01-02 07:48:44,780 INFO SystemMonitor:2914664 [interfaces.py:start():188] Started network monitoring 2025-01-02 07:48:44,843 DEBUG HandlerThread:2914664 [system_info.py:probe():152] Probing system 2025-01-02 07:48:44,847 DEBUG HandlerThread:2914664 [system_info.py:_probe_git():137] Probing git 2025-01-02 07:48:44,875 DEBUG HandlerThread:2914664 [system_info.py:_probe_git():145] Probing git done 2025-01-02 07:48:44,876 DEBUG HandlerThread:2914664 [system_info.py:probe():200] Probing system done 2025-01-02 07:48:44,876 DEBUG HandlerThread:2914664 [system_monitor.py:probe():223] {'os': 'Linux-6.8.0-48-generic-x86_64-with-glibc2.39', 'python': '3.10.0', 'heartbeatAt': '2025-01-02T07:48:44.843435', 'startedAt': '2025-01-02T07:48:44.192786', 'docker': None, 'cuda': None, 'args': ('--model_family', 'llama', '--apply_instruct_masks', '--token_scaled_loss', '--seq_parallel_size', '8', '--report_to', 'wandb', '--do_train', '--model_name_or_path', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--config_name', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--tokenizer_name', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--run_name', '_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_', '--output_dir', '/scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_', '--config_overrides_json', '', '--gradient_accumulation_steps', '2', '--per_device_train_batch_size', '1', '--bf16', '--learning_rate', '1e-6', '--min_lr_ratio', '0.1', '--lr_scheduler_type', 'cosine', '--max_grad_norm', '1.0', '--adam_beta1', '0.9', '--adam_beta2', '0.95', '--weight_decay', '0.1', '--warmup_ratio', '0.05', '--optim', 'adamw_torch', '--logging_steps', '1', '--log_level', 'info', '--save_steps', '200', '--dataloader_num_workers', '1', '--disable_tqdm', 'true', '--use_fast_tokenizer', 'false', '--remove_unused_columns', 'false', '--ddp_find_unused_parameters', 'false', '--fsdp', 'auto_wrap offload', '--gradient_checkpointing', '--tokenized_mds_train', '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/data/ft/bookclaim_balanced_pack_complete', '--cuda_empty_cache', '--num_train_epochs', '1'), 'state': 'running', 'program': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/prolong-final/finetune.py', 'codePathLocal': 'finetune.py', 'codePath': 'prolong-final/finetune.py', 'git': {'remote': 'https://github.com/chtmp223/BookGen-dev.git', 'commit': '9c36a7f300815c07d8a1bbceedfa1ef3781014f6'}, 'email': 'ctpham@umass.edu', 'root': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev', 'host': 'gpu018', 'username': 'ctpham_umass_edu', 'executable': '/scratch3/workspace/ctpham_umass_edu-ft/envs/prolong-final/bin/python3.10', 'cpu_count': 112, 'cpu_count_logical': 112, 'cpu_freq': {'current': 912.7720714285714, 'min': 800.0, 'max': 3800.0}, 'cpu_freq_per_core': [{'current': 794.049, 'min': 800.0, 'max': 3800.0}, {'current': 784.586, 'min': 800.0, 'max': 3800.0}, {'current': 790.761, 'min': 800.0, 'max': 3800.0}, {'current': 799.626, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 784.119, 'min': 800.0, 'max': 3800.0}, {'current': 787.971, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 797.352, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1340.4, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1500.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1771.582, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1500.0, 'min': 800.0, 'max': 3800.0}, {'current': 1252.363, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 791.339, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 791.68, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 784.633, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.458, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 785.754, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 789.093, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}], 'disk': {'/': {'total': 438.487850189209, 'used': 16.624420166015625}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 2015.3287467956543}} 2025-01-02 07:48:44,876 INFO HandlerThread:2914664 [system_monitor.py:probe():224] Finished collecting system info 2025-01-02 07:48:44,876 INFO HandlerThread:2914664 [system_monitor.py:probe():227] Publishing system info 2025-01-02 07:48:44,876 DEBUG HandlerThread:2914664 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2025-01-02 07:48:45,637 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/conda-environment.yaml 2025-01-02 07:48:51,906 DEBUG HandlerThread:2914664 [system_info.py:_save_conda():224] Saving conda packages done 2025-01-02 07:48:51,910 INFO HandlerThread:2914664 [system_monitor.py:probe():229] Finished publishing system info 2025-01-02 07:48:51,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:48:51,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: keepalive 2025-01-02 07:48:51,916 DEBUG SenderThread:2914664 [sender.py:send():379] send: files 2025-01-02 07:48:51,916 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-02 07:48:52,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: python_packages 2025-01-02 07:48:52,232 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: python_packages 2025-01-02 07:48:52,234 DEBUG SenderThread:2914664 [sender.py:send():379] send: telemetry 2025-01-02 07:48:52,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:48:52,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:48:52,235 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:48:52,289 INFO wandb-upload_0:2914664 [upload_job.py:push():130] Uploaded file /tmp/tmp19pvl77kwandb/cb2q4hzc-wandb-metadata.json 2025-01-02 07:48:52,374 DEBUG SenderThread:2914664 [sender.py:send():379] send: config 2025-01-02 07:48:52,376 DEBUG SenderThread:2914664 [sender.py:send():379] send: telemetry 2025-01-02 07:48:52,377 DEBUG SenderThread:2914664 [sender.py:send():379] send: metric 2025-01-02 07:48:52,377 DEBUG SenderThread:2914664 [sender.py:send():379] send: telemetry 2025-01-02 07:48:52,377 DEBUG SenderThread:2914664 [sender.py:send():379] send: metric 2025-01-02 07:48:52,377 WARNING SenderThread:2914664 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2025-01-02 07:48:52,377 DEBUG SenderThread:2914664 [sender.py:send():379] send: telemetry 2025-01-02 07:48:52,377 DEBUG SenderThread:2914664 [sender.py:send():379] send: telemetry 2025-01-02 07:48:52,377 DEBUG SenderThread:2914664 [sender.py:send():379] send: config 2025-01-02 07:48:52,640 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/conda-environment.yaml 2025-01-02 07:48:52,641 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:48:52,641 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-metadata.json 2025-01-02 07:48:52,641 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/requirements.txt 2025-01-02 07:48:53,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:48:54,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:48:54,642 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:48:55,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:48:55,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:48:56,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:48:57,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:48:58,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:48:59,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:00,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:00,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:01,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:02,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:03,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:04,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:05,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:05,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:06,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:06,652 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:49:07,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:49:07,233 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:49:07,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:08,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:09,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:10,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:10,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:11,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:12,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:13,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:14,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:15,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:16,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:16,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:16,660 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:49:16,661 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/config.yaml 2025-01-02 07:49:17,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:18,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:19,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:20,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:21,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:21,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:22,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:49:22,233 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:49:22,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:23,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:24,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:25,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:26,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:26,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:27,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:28,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:29,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:30,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:31,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:32,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:32,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:33,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:34,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:35,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:36,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:36,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:49:36,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: metric 2025-01-02 07:49:36,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: metric 2025-01-02 07:49:36,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: metric 2025-01-02 07:49:36,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: metric 2025-01-02 07:49:36,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: metric 2025-01-02 07:49:36,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:49:36,880 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:49:36,885 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:49:37,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:37,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:37,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:49:37,289 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:49:37,689 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:49:38,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:38,691 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:49:39,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:40,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:41,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:42,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:43,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:43,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:44,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:44,780 DEBUG SystemMonitor:2914664 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-02 07:49:44,783 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:49:45,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:46,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:47,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:48,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:48,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:48,699 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/config.yaml 2025-01-02 07:49:49,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:50,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:51,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:52,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:49:52,280 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:49:52,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:52,702 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:49:53,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:54,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:54,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:49:55,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:56,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:57,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:58,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:49:59,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:00,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:00,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:01,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:02,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:03,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:04,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:05,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:05,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:06,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:07,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:50:07,281 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:50:07,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:08,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:09,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:10,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:10,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:10,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:50:10,580 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:50:10,581 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:50:10,586 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:50:10,729 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:50:11,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:12,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:12,731 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:50:13,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:14,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:14,785 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:50:15,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:15,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:16,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:17,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:18,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:19,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:20,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:21,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:21,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:22,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:50:22,282 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:50:22,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:23,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:24,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:25,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:26,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:26,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:26,741 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:50:27,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:28,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:29,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:30,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:31,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:32,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:32,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:33,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:34,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:35,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:36,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:37,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:50:37,282 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:50:37,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:37,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:38,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:39,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:40,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:41,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:42,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:43,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:43,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:44,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:50:44,203 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:50:44,204 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:50:44,205 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:50:44,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:44,760 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:50:44,760 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:50:44,788 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:50:45,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:46,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:46,761 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:50:47,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:48,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:49,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:49,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:50,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:51,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:52,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:50:52,282 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:50:52,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:53,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:54,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:54,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:50:55,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:56,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:56,768 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:50:57,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:58,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:50:59,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:00,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:00,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:01,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:02,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:03,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:04,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:05,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:05,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:06,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:07,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:51:07,293 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:51:07,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:08,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:09,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:10,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:10,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:11,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:12,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:51:12,151 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:51:12,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:51:12,157 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:51:12,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:12,790 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:51:12,790 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:51:13,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:14,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:14,792 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:51:15,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:15,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:16,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:17,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:18,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:19,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:20,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:21,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:21,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:22,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:51:22,288 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:51:22,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:23,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:24,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:24,799 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:51:25,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:26,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:26,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:27,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:28,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:29,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:30,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:31,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:32,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:32,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:33,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:34,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:35,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:36,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:37,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:51:37,291 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:51:37,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:37,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:38,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:39,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:51:39,340 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:51:39,341 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:51:39,345 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:51:39,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:39,820 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:51:40,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:40,821 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:51:41,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:42,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:43,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:43,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:44,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:44,795 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:51:45,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:46,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:47,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:48,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:48,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:49,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:50,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:51,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:52,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:51:52,291 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:51:52,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:52,829 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:51:53,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:53,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:54,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:55,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:56,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:57,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:58,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:51:59,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:51:59,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:00,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:01,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:02,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:03,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:04,300 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:04,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:05,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:06,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:07,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:52:07,292 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:52:07,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:08,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:09,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:52:09,224 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:52:09,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:52:09,230 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:52:09,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:09,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:09,850 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:52:10,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:10,851 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:52:11,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:12,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:13,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:14,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:14,799 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:52:14,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:15,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:16,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:17,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:18,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:19,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:20,300 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:20,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:21,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:22,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:52:22,293 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:52:22,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:22,859 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:52:23,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:24,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:25,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:25,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:26,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:27,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:28,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:29,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:30,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:31,300 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:31,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:32,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:33,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:34,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:35,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:36,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:36,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:37,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:52:37,293 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:52:37,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:38,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:39,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:52:39,103 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:52:39,104 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:52:39,109 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:52:39,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:39,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:52:40,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:40,882 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:52:41,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:41,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:42,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:43,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:44,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:44,804 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:52:45,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:46,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:46,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:47,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:48,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:49,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:50,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:51,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:52,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:52:52,293 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:52:52,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:52,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:53,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:54,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:55,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:56,892 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:52:56,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:57,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:58,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:52:58,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:52:59,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:00,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:01,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:02,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:03,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:04,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:04,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:05,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:06,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:07,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:53:07,293 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:53:07,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:08,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:09,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:09,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:10,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:53:10,843 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:53:10,844 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:53:10,846 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:53:10,920 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:53:11,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:12,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:12,921 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:53:13,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:14,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:14,807 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:53:14,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:15,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:16,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:17,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:18,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:19,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:20,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:20,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:21,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:22,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:22,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:53:22,293 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:53:23,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:24,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:24,929 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:53:25,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:26,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:26,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:27,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:28,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:29,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:30,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:31,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:31,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:32,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:33,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:34,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:35,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:36,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:36,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:37,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:37,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:53:37,297 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:53:38,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:39,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:40,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:53:40,230 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:53:40,230 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:53:40,232 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:53:40,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:40,951 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:53:40,952 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:53:41,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:41,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:42,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:43,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:44,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:44,813 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:53:45,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:46,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:46,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:47,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:48,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:49,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:50,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:51,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:52,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:53:52,297 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:53:52,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:52,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:53,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:54,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:54,960 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:53:55,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:56,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:57,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:58,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:53:58,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:53:59,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:00,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:01,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:02,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:03,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:04,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:04,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:05,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:06,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:07,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:54:07,306 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:54:07,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:08,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:09,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:09,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:10,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:10,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:54:10,751 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:54:10,751 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:54:10,753 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:54:10,978 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:54:10,979 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:54:11,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:12,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:12,980 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:54:13,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:14,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:14,815 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:54:14,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:15,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:16,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:17,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:18,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:19,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:20,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:20,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:21,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:22,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:54:22,303 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:54:22,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:23,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:24,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:24,989 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:54:25,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:25,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:26,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:27,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:28,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:29,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:30,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:31,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:31,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:32,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:33,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:34,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:35,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:36,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:36,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:37,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:54:37,307 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:54:37,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:38,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:39,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:40,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:40,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:54:40,903 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:54:40,903 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:54:40,906 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:54:41,012 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:54:41,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:42,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:42,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:43,014 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:54:43,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:44,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:44,820 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:54:45,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:46,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:47,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:47,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:48,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:49,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:50,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:51,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:52,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:54:52,304 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:54:52,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:52,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:53,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:54,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:55,023 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:54:55,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:56,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:57,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:58,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:54:58,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:54:59,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:00,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:01,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:02,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:03,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:03,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:04,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:05,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:06,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:07,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:55:07,312 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:55:07,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:08,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:08,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:09,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:10,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:10,514 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:55:10,516 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:55:10,516 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:55:10,518 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:55:11,049 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:55:11,049 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:55:11,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:12,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:13,050 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:55:13,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:14,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:14,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:14,824 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:55:15,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:16,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:17,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:18,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:19,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:20,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:20,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:21,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:22,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:55:22,307 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:55:22,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:23,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:24,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:25,059 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:55:25,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:25,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:26,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:27,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:28,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:29,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:30,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:31,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:31,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:32,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:33,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:34,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:35,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:36,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:36,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:37,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:55:37,312 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:55:37,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:38,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:39,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:40,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:41,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:55:41,264 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:55:41,265 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:55:41,270 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:55:41,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:41,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:42,080 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:55:42,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:43,081 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:55:43,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:44,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:44,828 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:55:45,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:46,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:46,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:47,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:48,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:49,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:50,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:51,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:52,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:55:52,308 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:55:52,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:52,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:53,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:54,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:55,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:56,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:57,091 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:55:57,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:58,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:55:58,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:55:59,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:00,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:01,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:02,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:03,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:04,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:04,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:05,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:06,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:07,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:56:07,313 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:56:07,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:08,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:09,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:09,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:10,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:11,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:12,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:13,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:56:13,002 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:56:13,002 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:56:13,005 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:56:13,107 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:56:13,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:14,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:14,833 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:56:14,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:15,109 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:56:15,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:16,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:17,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:18,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:19,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:20,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:20,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:21,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:22,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:56:22,308 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:56:22,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:23,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:24,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:25,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:25,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:26,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:27,118 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:56:27,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:28,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:29,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:30,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:31,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:31,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:32,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:33,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:34,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:35,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:36,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:36,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:37,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:56:37,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:56:37,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:38,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:39,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:40,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:41,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:41,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:42,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:43,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:44,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:56:44,402 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:56:44,402 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:56:44,407 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:56:44,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:44,836 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:56:45,139 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:56:45,139 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:56:45,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:46,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:46,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:47,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:48,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:49,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:50,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:51,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:52,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:56:52,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:56:52,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:52,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:53,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:54,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:55,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:56,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:57,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:58,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:56:58,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:56:59,151 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:56:59,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:00,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:01,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:02,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:03,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:04,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:04,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:05,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:06,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:07,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:57:07,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:57:07,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:08,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:09,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:09,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:10,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:11,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:12,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:13,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:14,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:14,837 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:57:14,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:15,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:57:15,221 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:57:15,222 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:57:15,224 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:57:15,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:16,170 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:57:16,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:17,171 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:57:17,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:18,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:19,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:20,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:20,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:21,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:22,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:57:22,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:57:22,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:23,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:24,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:25,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:25,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:26,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:27,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:29,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:30,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:31,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:31,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:31,181 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:57:32,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:33,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:34,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:35,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:36,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:36,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:37,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:37,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:57:37,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:57:38,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:39,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:40,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:41,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:41,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:42,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:43,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:44,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:44,845 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:57:45,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:46,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:46,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:57:46,322 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:57:46,322 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:57:46,323 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:57:46,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:47,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:47,196 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:57:47,197 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:57:48,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:49,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:49,198 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:57:50,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:51,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:52,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:52,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:57:52,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:57:52,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:53,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:54,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:55,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:56,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:57,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:58,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:57:58,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:57:59,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:00,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:01,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:02,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:03,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:03,209 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:58:04,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:04,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:05,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:06,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:07,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:07,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:58:07,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:58:08,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:09,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:09,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:10,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:11,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:12,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:13,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:14,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:14,854 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:58:14,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:15,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:16,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:17,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:18,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:19,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:20,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:20,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:20,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:58:20,593 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:58:20,593 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:58:20,595 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:58:21,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:21,228 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:58:21,229 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:58:22,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:22,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:58:22,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:58:23,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:23,230 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:58:24,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:25,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:25,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:26,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:27,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:28,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:29,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:30,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:31,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:31,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:32,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:33,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:33,236 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:58:34,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:35,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:36,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:36,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:37,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:37,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:58:37,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:58:38,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:39,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:40,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:41,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:42,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:42,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:43,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:44,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:44,858 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:58:45,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:46,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:47,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:47,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:48,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:48,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:58:48,180 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:58:48,181 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:58:48,182 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:58:48,253 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:58:49,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:49,254 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:58:50,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:51,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:52,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:52,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:58:52,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:58:52,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:53,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:54,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:55,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:56,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:57,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:58,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:58:58,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:58:59,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:00,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:01,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:02,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:03,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:03,266 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:59:03,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:04,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:05,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:06,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:07,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:07,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:59:07,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:59:08,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:08,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:09,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:10,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:11,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:12,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:13,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:14,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:14,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:14,861 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:59:15,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:16,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:17,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:59:17,031 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:59:17,031 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:59:17,033 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:59:17,295 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:59:17,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:18,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:19,296 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:59:19,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:19,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:20,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:21,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:22,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:59:22,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:59:22,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:23,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:24,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:25,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:25,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:26,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:27,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:28,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:29,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:30,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:31,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:31,304 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:59:31,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:32,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:33,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:34,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:35,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:36,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:36,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:37,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:59:37,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:59:37,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:38,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:39,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:40,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:41,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:42,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:42,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:43,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:44,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:44,858 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 07:59:45,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:46,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:47,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:47,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:47,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 07:59:47,941 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 07:59:47,942 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 07:59:47,947 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 07:59:48,344 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 07:59:48,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:49,345 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 07:59:49,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:50,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:51,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:52,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 07:59:52,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 07:59:52,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:52,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:53,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:54,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:55,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:56,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:57,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 07:59:57,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:58,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 07:59:59,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:00,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:01,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:02,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:02,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:03,355 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:00:03,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:04,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:05,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:06,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:07,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:00:07,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:00:07,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:07,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:08,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:09,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:10,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:11,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:12,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:12,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:13,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:14,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:14,860 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:00:15,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:16,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:17,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:17,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:18,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:19,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:20,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:21,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:00:21,450 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:00:21,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:00:21,454 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:00:21,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:22,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:00:22,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:00:22,374 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:00:22,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:23,375 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:00:23,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:23,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:24,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:25,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:26,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:27,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:28,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:28,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:29,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:30,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:31,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:32,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:33,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:34,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:34,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:35,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:36,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:37,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:00:37,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:00:37,386 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:00:37,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:38,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:39,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:40,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:40,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:41,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:42,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:43,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:44,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:44,864 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:00:45,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:45,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:46,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:47,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:48,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:49,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:50,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:51,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:51,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:52,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:00:52,320 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:00:52,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:53,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:53,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:00:53,977 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:00:53,977 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:00:53,979 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:00:54,410 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:00:54,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:55,411 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:00:55,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:56,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:57,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:00:57,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:58,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:00:59,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:00,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:01,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:02,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:02,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:03,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:04,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:05,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:06,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:07,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:01:07,313 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:01:07,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:07,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:08,422 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:01:08,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:09,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:10,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:11,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:12,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:12,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:13,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:14,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:14,868 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:01:15,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:16,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:17,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:17,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:18,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:19,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:20,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:21,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:22,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:01:22,316 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:01:22,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:22,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:01:22,909 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:01:22,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:01:22,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:22,911 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:01:23,448 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:01:23,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:24,449 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:01:24,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:25,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:26,450 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:01:26,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:27,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:28,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:28,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:29,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:30,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:31,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:32,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:33,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:33,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:34,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:35,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:36,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:37,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:01:37,316 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:01:37,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:38,459 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:01:38,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:39,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:39,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:40,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:41,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:42,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:43,347 ERROR gpu :2914664 [interfaces.py:monitor():142] Failed to sample metric: Not Supported 2025-01-02 08:01:43,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:44,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:44,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:44,874 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:01:45,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:46,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:47,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:48,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:49,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:49,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:50,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:51,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:52,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:01:52,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:01:52,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:01:52,514 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:01:52,515 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:01:52,516 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:01:52,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:53,478 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:01:53,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:54,479 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:01:54,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:54,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:01:55,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:56,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:57,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:58,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:01:59,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:00,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:00,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:01,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:02,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:03,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:04,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:05,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:06,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:06,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:07,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:02:07,316 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:02:07,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:08,489 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:02:08,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:09,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:10,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:11,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:11,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:12,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:13,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:14,877 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:02:14,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:15,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:16,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:16,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:17,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:18,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:19,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:20,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:21,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:22,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:02:22,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:02:22,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:22,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:22,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:02:22,947 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:02:22,948 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:02:22,949 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:02:23,508 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:02:23,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:24,509 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:02:24,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:25,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:26,511 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:02:26,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:27,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:27,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:28,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:29,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:30,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:31,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:32,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:33,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:33,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:34,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:35,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:36,517 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:02:36,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:37,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:02:37,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:02:37,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:38,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:39,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:39,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:40,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:41,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:42,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:43,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:44,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:44,881 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:02:44,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:45,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:46,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:47,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:48,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:49,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:49,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:50,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:02:50,118 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:02:50,118 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:02:50,120 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:02:50,534 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:02:50,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:51,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:52,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:02:52,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:02:52,536 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:02:52,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:53,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:54,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:55,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:02:55,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:56,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:57,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:58,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:02:59,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:00,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:01,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:01,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:02,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:03,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:04,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:05,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:06,546 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:03:06,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:06,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:07,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:03:07,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:03:07,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:08,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:09,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:10,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:11,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:12,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:12,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:13,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:14,885 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:03:14,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:15,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:16,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:17,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:17,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:18,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:19,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:20,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:21,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:03:21,455 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:03:21,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:03:21,457 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:03:21,560 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:03:21,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:22,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:03:22,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:03:22,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:22,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:23,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:24,563 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:03:24,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:25,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:26,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:27,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:28,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:28,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:29,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:30,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:31,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:33,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:34,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:34,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:34,570 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:03:35,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:36,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:37,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:37,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:03:37,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:03:38,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:39,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:39,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:40,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:41,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:42,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:43,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:44,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:44,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:44,895 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:03:45,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:46,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:47,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:48,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:49,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:49,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:49,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:03:49,949 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:03:49,950 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:03:49,952 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:03:50,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:50,597 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:03:51,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:52,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:52,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:03:52,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:03:52,599 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:03:53,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:54,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:55,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:55,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:03:56,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:57,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:58,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:03:59,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:00,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:01,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:01,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:02,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:03,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:04,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:05,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:06,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:06,609 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:04:07,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:07,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:04:07,317 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:04:07,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:08,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:09,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:10,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:11,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:12,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:12,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:13,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:14,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:14,893 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:04:15,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:16,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:17,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:17,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:18,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:19,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:20,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:21,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:21,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:04:21,823 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:04:21,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:04:21,826 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:04:22,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:22,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:04:22,318 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:04:22,631 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:04:23,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:23,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:24,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:24,632 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:04:25,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:26,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:27,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:28,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:29,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:29,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:30,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:31,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:32,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:33,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:34,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:35,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:35,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:36,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:37,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:37,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:04:37,318 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:04:38,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:38,643 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:04:39,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:40,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:40,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:41,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:42,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:43,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:44,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:44,896 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:04:45,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:45,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:46,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:47,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:48,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:49,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:50,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:51,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:51,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:52,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:52,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:04:52,237 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:04:52,237 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:04:52,240 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:04:52,663 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:04:52,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:04:52,699 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:04:53,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:54,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:54,664 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:04:55,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:56,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:56,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:04:57,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:58,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:04:59,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:00,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:01,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:02,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:02,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:03,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:04,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:05,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:06,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:06,673 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:05:07,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:07,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:05:07,381 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:05:08,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:08,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:09,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:10,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:11,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:12,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:13,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:14,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:14,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:14,900 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:05:15,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:16,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:17,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:18,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:19,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:19,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:20,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:21,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:22,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:22,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:05:22,385 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:05:23,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:05:23,016 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:05:23,016 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:05:23,019 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:05:23,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:23,703 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:05:24,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:24,704 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:05:25,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:25,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:26,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:26,706 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:05:27,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:28,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:29,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:30,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:30,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:31,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:32,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:33,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:34,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:35,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:36,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:36,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:37,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:05:37,385 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:05:37,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:38,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:38,715 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:05:39,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:40,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:41,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:41,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:42,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:43,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:44,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:44,904 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:05:45,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:46,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:46,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:47,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:48,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:49,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:50,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:51,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:52,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:05:52,388 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:05:52,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:52,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:53,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:05:53,323 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:05:53,324 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:05:53,329 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:05:53,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:53,733 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:05:54,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:55,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:56,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:56,735 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:05:57,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:57,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:05:58,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:05:59,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:00,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:01,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:02,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:03,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:03,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:04,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:05,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:06,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:07,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:06:07,388 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:06:07,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:08,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:08,744 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:06:09,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:09,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:10,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:11,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:12,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:13,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:14,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:14,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:14,908 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:06:15,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:16,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:17,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:18,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:19,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:19,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:20,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:20,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:06:20,984 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:06:20,984 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:06:20,985 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:06:21,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:21,764 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:06:22,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:06:22,388 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:06:22,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:22,766 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:06:23,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:24,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:24,767 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:06:25,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:25,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:26,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:27,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:28,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:29,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:30,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:30,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:31,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:32,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:33,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:34,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:35,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:35,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:36,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:36,776 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:06:37,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:06:37,389 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:06:37,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:38,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:39,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:40,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:41,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:41,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:42,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:43,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:44,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:44,913 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:06:45,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:46,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:46,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:47,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:48,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:49,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:50,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:51,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:52,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:06:52,399 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:06:52,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:52,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:53,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:06:53,511 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:06:53,512 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:06:53,513 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:06:53,797 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:06:53,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:54,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:55,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:56,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:56,799 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:06:57,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:57,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:06:58,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:06:59,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:00,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:01,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:02,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:03,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:03,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:04,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:05,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:06,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:06,807 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:07:07,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:07:07,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:07:07,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:08,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:09,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:09,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:10,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:11,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:12,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:13,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:14,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:14,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:14,914 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:07:15,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:16,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:17,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:18,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:19,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:07:19,641 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:07:19,641 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:07:19,643 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:07:19,826 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:07:20,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:20,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:21,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:22,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:22,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:07:22,392 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:07:22,828 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:07:23,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:24,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:25,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:25,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:26,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:27,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:28,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:29,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:30,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:31,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:31,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:32,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:33,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:34,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:34,836 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:07:35,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:36,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:36,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:37,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:37,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:07:37,392 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:07:38,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:39,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:40,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:41,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:42,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:42,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:43,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:44,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:44,918 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:07:45,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:46,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:47,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:47,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:48,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:49,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:49,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:07:49,678 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:07:49,678 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:07:49,679 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:07:49,858 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:07:50,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:51,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:52,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:52,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:07:52,392 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:07:52,860 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:07:53,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:53,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:07:54,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:55,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:56,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:57,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:58,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:59,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:07:59,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:00,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:01,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:02,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:03,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:04,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:04,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:04,869 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:08:05,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:06,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:07,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:07,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:08:07,392 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:08:08,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:09,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:09,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:10,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:11,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:12,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:13,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:14,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:14,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:14,928 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:08:15,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:16,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:17,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:18,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:19,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:19,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:08:19,559 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:08:19,560 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:08:19,561 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:08:19,891 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:08:19,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:20,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:21,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:22,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:22,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:08:22,392 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:08:22,894 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:08:23,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:24,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:25,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:25,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:26,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:27,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:28,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:29,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:30,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:30,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:31,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:32,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:33,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:34,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:34,902 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:08:35,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:35,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:36,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:37,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:37,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:08:37,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:08:38,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:39,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:40,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:41,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:41,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:42,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:43,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:44,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:44,930 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:08:45,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:46,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:46,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:47,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:48,105 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:48,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:08:48,912 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:08:48,913 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:08:48,914 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:08:48,925 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:08:49,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:50,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:50,928 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:08:51,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:52,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:52,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:52,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:08:52,392 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:08:53,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:54,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:55,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:56,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:57,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:57,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:08:58,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:08:59,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:00,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:01,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:02,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:02,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:03,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:04,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:04,937 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:09:05,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:06,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:07,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:07,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:09:07,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:09:08,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:08,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:09,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:10,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:11,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:12,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:13,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:13,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:14,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:14,935 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:09:15,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:16,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:17,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:18,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:18,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:19,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:19,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:09:19,337 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:09:19,337 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:09:19,339 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:09:19,953 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:09:20,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:20,954 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:09:21,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:22,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:22,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:09:22,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:09:22,956 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:09:23,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:24,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:24,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:25,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:26,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:27,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:28,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:29,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:29,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:30,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:31,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:32,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:33,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:34,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:35,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:35,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:36,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:36,965 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:09:37,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:09:37,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:09:37,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:38,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:39,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:40,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:40,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:41,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:42,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:43,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:44,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:44,933 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:09:45,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:45,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:46,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:47,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:48,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:49,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:50,492 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:51,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:51,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:52,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:09:52,401 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:09:52,493 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:53,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:54,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:55,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:09:55,266 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:09:55,266 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:09:55,268 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:09:55,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:55,991 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:09:56,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:56,992 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:09:57,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:57,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:09:58,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:09:59,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:00,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:01,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:02,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:02,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:03,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:04,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:05,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:06,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:07,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:10:07,398 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:10:07,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:08,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:08,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:09,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:10,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:11,003 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:10:11,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:12,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:13,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:13,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:14,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:14,935 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:10:15,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:16,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:17,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:18,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:19,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:19,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:20,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:21,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:22,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:10:22,402 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:10:22,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:23,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:24,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:24,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:25,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:26,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:26,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:10:26,912 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:10:26,912 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:10:26,914 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:10:27,020 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:10:27,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:28,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:29,021 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:10:29,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:30,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:30,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:31,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:32,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:33,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:34,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:35,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:36,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:36,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:37,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:10:37,402 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:10:37,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:38,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:39,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:40,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:41,029 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:10:41,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:42,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:42,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:43,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:44,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:44,939 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:10:45,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:46,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:47,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:47,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:48,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:49,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:50,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:51,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:52,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:10:52,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:10:52,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:52,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:53,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:54,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:55,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:56,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:56,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:10:56,884 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:10:56,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:10:56,885 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:10:57,043 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:10:57,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:58,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:10:58,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:10:59,044 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:10:59,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:00,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:01,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:02,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:03,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:04,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:04,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:05,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:06,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:07,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:11:07,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:11:07,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:08,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:09,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:10,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:10,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:11,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:12,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:13,053 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:11:13,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:14,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:14,943 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:11:15,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:15,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:16,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:17,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:18,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:19,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:20,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:21,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:21,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:22,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:11:22,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:11:22,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:23,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:24,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:25,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:26,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:26,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:27,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:28,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:29,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:11:29,721 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:11:29,722 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:11:29,723 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:11:30,077 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:11:30,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:30,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:31,078 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:11:31,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:32,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:32,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:33,080 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:11:33,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:34,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:35,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:36,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:37,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:11:37,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:11:37,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:37,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:38,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:39,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:40,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:41,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:42,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:42,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:43,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:44,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:44,948 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:11:45,089 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:11:45,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:46,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:47,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:47,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:48,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:49,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:50,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:51,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:52,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:11:52,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:11:52,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:53,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:53,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:54,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:55,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:56,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:57,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:58,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:11:58,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:11:59,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:00,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:00,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:12:00,964 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:12:00,965 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:12:00,967 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:12:01,112 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:12:01,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:02,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:03,113 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:12:03,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:04,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:04,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:05,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:06,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:07,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:12:07,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:12:07,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:08,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:09,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:10,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:10,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:11,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:12,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:13,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:14,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:14,951 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:12:15,124 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:12:15,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:15,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:16,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:17,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:18,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:19,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:20,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:21,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:21,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:22,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:12:22,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:12:22,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:23,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:24,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:25,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:26,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:26,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:27,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:28,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:29,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:12:29,796 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:12:29,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:12:29,797 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:12:30,148 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:12:30,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:31,149 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:12:31,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:32,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:32,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:33,150 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:12:33,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:34,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:35,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:36,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:37,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:12:37,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:12:37,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:37,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:38,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:39,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:40,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:41,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:42,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:42,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:43,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:44,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:44,956 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:12:45,158 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:12:45,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:46,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:47,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:47,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:48,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:49,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:50,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:51,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:52,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:12:52,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:12:52,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:53,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:53,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:54,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:55,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:56,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:57,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:58,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:58,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:12:59,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:12:59,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:12:59,776 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:12:59,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:12:59,778 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:13:00,180 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:13:00,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:01,180 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:13:01,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:02,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:03,182 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:13:03,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:04,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:04,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:05,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:06,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:07,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:13:07,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:13:07,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:08,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:09,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:10,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:10,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:11,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:12,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:13,187 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:13:13,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:14,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:14,960 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:13:15,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:15,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:16,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:17,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:18,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:19,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:20,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:21,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:21,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:22,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:13:22,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:13:22,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:23,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:24,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:25,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:26,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:26,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:27,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:28,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:29,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:13:29,050 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:13:29,050 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:13:29,052 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:13:29,214 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:13:30,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:31,217 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:13:31,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:32,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:32,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:33,218 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:13:33,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:34,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:35,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:36,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:37,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:37,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:13:37,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:13:37,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:38,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:39,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:40,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:41,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:42,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:42,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:43,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:44,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:44,964 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:13:45,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:46,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:47,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:47,227 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:13:47,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:48,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:49,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:50,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:51,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:52,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:52,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:13:52,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:13:53,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:53,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:54,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:55,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:56,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:57,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:58,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:13:58,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:13:59,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:00,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:01,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:02,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:03,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:03,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:14:03,753 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:14:03,754 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:14:03,755 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:14:03,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:04,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:04,247 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:14:05,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:05,248 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:14:06,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:07,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:07,251 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:14:07,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:14:07,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:14:08,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:09,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:09,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:10,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:11,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:12,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:13,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:14,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:14,968 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:14:14,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:15,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:16,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:17,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:17,257 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:14:18,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:19,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:20,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:20,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:21,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:22,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:22,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:14:22,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:14:23,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:24,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:25,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:26,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:26,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:27,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:28,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:29,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:30,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:31,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:31,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:32,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:33,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:14:33,095 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:14:33,095 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:14:33,097 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:14:33,284 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:14:33,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:34,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:35,285 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:14:35,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:36,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:37,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:14:37,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:14:37,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:37,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:38,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:39,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:40,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:41,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:42,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:42,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:43,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:44,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:44,972 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:14:45,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:46,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:47,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:47,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:48,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:49,294 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:14:49,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:50,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:51,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:52,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:14:52,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:14:52,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:53,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:53,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:54,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:55,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:56,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:57,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:58,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:14:58,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:14:59,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:00,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:01,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:02,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:03,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:03,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:03,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:15:03,858 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:15:03,859 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:15:03,860 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:15:04,317 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:15:04,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:05,318 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:15:05,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:06,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:07,319 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:15:07,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:15:07,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:15:07,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:08,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:09,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:09,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:10,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:11,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:12,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:13,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:14,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:14,973 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:15:14,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:15,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:16,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:17,326 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:15:17,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:18,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:19,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:20,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:20,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:21,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:22,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:15:22,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:15:22,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:23,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:24,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:25,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:25,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:26,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:27,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:28,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:29,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:30,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:30,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:31,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:31,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:15:31,834 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:15:31,835 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:15:31,836 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:15:32,358 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:15:32,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:33,359 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:15:33,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:34,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:35,361 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:15:35,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:36,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:36,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:37,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:15:37,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:15:37,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:38,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:39,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:40,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:41,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:42,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:42,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:43,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:44,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:44,977 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:15:45,367 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:15:45,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:46,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:47,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:47,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:48,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:49,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:50,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:51,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:52,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:15:52,408 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:15:52,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:53,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:53,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:54,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:55,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:56,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:57,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:58,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:15:58,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:15:59,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:00,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:01,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:16:01,074 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:16:01,074 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:16:01,076 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:16:01,385 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:16:01,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:02,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:03,387 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:16:03,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:04,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:04,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:05,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:06,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:07,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:16:07,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:16:07,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:08,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:09,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:09,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:10,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:11,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:12,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:13,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:14,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:14,981 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:16:14,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:15,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:16,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:17,396 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:16:17,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:18,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:19,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:20,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:20,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:21,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:22,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:16:22,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:16:22,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:23,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:24,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:25,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:26,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:26,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:27,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:28,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:29,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:30,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:31,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:31,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:32,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:33,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:33,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:16:33,866 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:16:33,866 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:16:33,868 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:16:34,422 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:16:34,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:35,423 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:16:35,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:36,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:37,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:37,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:16:37,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:16:37,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:38,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:39,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:40,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:41,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:42,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:42,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:43,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:44,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:44,985 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:16:45,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:46,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:47,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:48,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:48,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:49,432 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:16:49,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:50,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:51,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:52,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:16:52,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:16:52,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:53,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:54,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:54,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:16:55,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:56,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:57,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:58,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:59,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:16:59,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:00,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:01,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:02,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:03,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:04,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:17:04,372 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:17:04,372 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:17:04,374 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:17:04,452 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:17:04,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:05,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:05,779 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:06,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:07,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:17:07,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:17:07,454 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:17:07,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:08,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:09,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:10,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:11,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:11,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:12,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:13,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:14,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:14,989 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:17:15,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:16,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:16,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:17,460 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:17:17,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:18,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:19,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:20,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:21,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:22,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:17:22,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:17:22,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:22,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:23,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:24,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:25,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:26,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:27,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:27,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:28,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:29,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:30,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:31,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:31,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:17:31,898 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:17:31,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:17:31,900 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:17:32,482 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:17:32,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:33,483 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:17:33,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:33,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:34,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:35,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:36,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:37,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:17:37,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:17:37,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:38,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:39,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:39,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:40,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:41,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:42,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:43,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:44,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:44,993 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:17:45,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:45,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:46,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:47,493 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:17:47,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:48,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:49,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:50,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:50,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:51,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:52,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:17:52,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:17:52,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:53,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:54,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:55,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:56,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:17:56,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:57,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:58,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:17:59,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:00,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:01,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:01,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:02,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:18:02,580 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:18:02,580 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:18:02,581 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:18:02,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:03,514 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:18:03,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:04,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:05,516 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:18:05,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:06,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:06,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:07,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:18:07,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:18:07,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:08,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:09,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:10,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:11,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:12,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:12,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:13,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:14,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:14,997 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:18:15,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:16,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:17,524 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:18:17,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:17,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:18,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:19,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:20,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:21,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:22,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:18:22,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:18:22,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:23,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:23,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:24,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:25,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:26,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:28,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:28,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:28,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:29,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:31,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:32,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:32,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:18:32,015 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:18:32,016 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:18:32,018 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:18:32,541 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:18:33,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:33,542 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:18:33,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:34,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:35,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:36,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:37,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:37,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:18:37,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:18:38,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:39,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:39,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:40,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:41,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:42,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:43,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:44,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:45,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:45,002 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:18:45,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:46,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:47,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:47,551 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:18:48,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:49,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:50,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:50,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:51,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:52,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:52,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:18:52,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:18:53,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:54,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:55,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:56,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:57,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:18:57,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:58,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:18:59,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:00,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:01,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:02,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:02,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:02,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:19:02,331 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:19:02,332 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:19:02,334 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:19:02,571 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:19:03,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:03,572 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:19:04,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:05,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:05,573 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:19:06,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:07,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:07,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:19:07,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:19:07,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:08,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:09,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:10,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:11,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:12,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:12,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:13,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:14,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:15,005 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:19:15,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:15,581 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:19:16,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:17,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:17,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:18,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:19,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:20,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:21,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:22,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:22,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:19:22,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:19:23,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:23,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:24,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:25,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:26,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:27,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:28,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:29,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:29,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:30,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:30,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:19:30,474 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:19:30,474 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:19:30,475 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:19:30,603 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:19:31,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:31,604 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:19:32,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:33,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:33,606 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:19:34,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:35,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:35,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:36,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:37,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:37,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:19:37,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:19:38,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:39,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:40,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:41,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:41,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:42,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:43,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:44,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:45,008 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:19:45,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:45,615 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:19:46,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:46,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:47,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:48,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:49,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:50,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:51,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:51,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:52,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:52,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:19:52,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:19:53,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:54,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:55,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:56,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:19:56,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:57,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:58,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:19:59,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:00,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:00,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:20:00,339 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:20:00,339 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:20:00,340 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:20:00,634 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:20:01,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:01,635 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:20:01,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:02,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:03,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:03,637 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:20:04,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:05,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:06,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:07,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:07,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:07,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:20:07,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:20:08,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:09,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:10,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:11,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:12,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:13,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:13,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:14,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:15,012 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:20:15,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:15,645 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:20:16,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:17,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:18,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:18,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:19,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:20,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:21,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:22,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:22,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:20:22,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:20:23,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:24,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:24,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:25,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:26,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:27,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:28,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:29,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:29,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:30,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:31,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:31,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:20:31,553 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:20:31,553 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:20:31,554 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:20:31,667 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:20:32,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:33,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:33,669 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:20:34,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:35,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:35,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:36,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:37,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:37,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:20:37,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:20:38,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:39,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:40,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:40,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:41,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:42,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:43,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:44,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:45,017 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:20:45,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:45,678 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:20:46,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:46,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:47,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:48,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:49,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:50,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:51,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:51,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:52,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:52,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:20:52,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:20:53,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:54,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:55,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:56,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:20:56,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:57,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:58,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:20:59,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:00,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:01,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:21:01,003 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:21:01,004 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:21:01,006 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:21:01,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:01,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:01,698 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:21:02,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:03,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:03,699 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:21:04,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:05,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:06,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:07,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:07,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:07,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:21:07,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:21:08,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:09,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:10,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:11,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:12,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:13,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:13,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:14,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:15,021 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:21:15,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:15,707 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:21:16,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:17,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:18,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:18,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:19,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:20,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:21,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:22,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:21:22,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:21:22,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:23,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:24,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:24,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:25,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:26,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:27,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:28,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:29,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:29,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:29,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:21:29,435 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:21:29,435 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:21:29,437 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:21:29,727 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:21:30,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:31,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:31,729 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:21:32,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:33,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:34,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:35,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:35,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:36,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:37,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:21:37,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:21:37,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:38,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:39,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:40,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:41,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:41,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:42,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:43,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:44,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:45,025 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:21:45,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:46,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:47,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:47,741 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:21:47,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:48,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:49,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:50,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:51,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:52,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:52,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:21:52,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:21:52,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:53,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:54,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:55,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:56,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:57,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:21:57,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:58,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:21:59,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:00,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:01,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:02,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:22:02,102 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:22:02,102 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:22:02,104 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:22:02,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:02,761 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:22:02,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:03,762 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:22:03,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:04,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:05,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:06,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:07,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:22:07,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:22:07,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:07,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:08,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:09,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:10,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:11,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:12,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:13,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:13,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:14,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:15,029 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:22:15,770 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:22:15,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:16,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:17,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:18,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:18,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:19,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:20,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:21,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:22,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:22:22,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:22:22,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:23,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:24,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:24,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:25,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:26,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:27,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:28,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:29,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:29,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:30,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:31,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:22:31,800 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:22:31,800 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:22:31,801 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:22:32,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:32,792 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:22:32,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:33,793 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:22:33,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:34,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:35,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:35,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:36,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:37,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:22:37,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:22:37,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:38,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:39,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:40,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:41,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:41,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:42,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:44,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:45,033 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:22:45,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:45,800 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:22:46,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:47,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:47,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:48,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:49,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:50,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:51,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:52,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:52,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:52,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:22:52,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:22:53,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:54,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:55,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:56,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:57,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:57,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:22:58,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:22:59,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:00,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:01,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:01,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:23:01,381 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:23:01,381 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:23:01,383 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:23:01,821 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:23:02,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:02,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:03,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:03,822 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:23:04,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:05,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:06,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:07,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:07,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:23:07,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:23:08,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:08,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:09,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:10,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:11,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:12,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:13,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:14,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:14,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:15,037 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:23:15,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:16,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:17,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:17,832 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:23:18,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:19,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:19,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:20,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:21,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:22,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:22,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:23:22,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:23:23,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:24,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:24,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:25,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:26,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:27,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:28,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:29,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:29,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:30,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:31,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:32,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:32,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:23:32,999 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:23:32,999 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:23:33,001 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:23:33,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:33,853 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:23:34,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:34,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:35,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:35,854 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:23:36,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:37,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:37,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:23:37,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:23:38,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:39,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:40,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:40,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:41,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:42,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:43,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:44,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:45,041 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:23:45,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:46,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:46,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:47,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:47,862 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:23:48,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:49,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:50,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:51,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:51,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:52,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:52,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:23:52,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:23:53,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:54,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:55,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:56,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:57,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:23:57,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:58,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:23:59,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:00,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:01,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:02,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:02,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:02,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:24:02,644 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:24:02,644 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:24:02,645 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:24:02,892 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:24:03,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:03,894 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:24:04,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:05,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:05,896 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:24:06,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:07,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:07,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:24:07,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:24:07,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:08,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:09,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:10,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:11,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:12,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:13,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:13,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:14,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:15,045 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:24:15,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:15,904 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:24:16,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:17,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:18,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:18,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:19,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:20,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:21,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:22,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:22,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:24:22,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:24:23,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:24,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:24,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:25,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:26,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:27,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:28,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:29,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:29,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:30,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:31,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:32,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:32,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:24:32,415 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:24:32,415 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:24:32,416 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:24:32,935 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:24:33,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:33,936 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:24:34,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:35,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:35,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:36,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:37,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:24:37,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:24:37,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:38,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:39,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:40,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:41,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:41,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:42,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:43,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:44,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:45,050 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:24:45,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:46,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:46,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:47,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:47,945 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:24:48,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:49,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:50,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:51,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:52,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:52,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:24:52,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:24:52,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:53,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:54,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:55,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:56,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:57,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:58,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:24:58,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:24:59,419 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:00,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:01,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:02,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:03,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:03,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:04,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:04,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:25:04,897 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:25:04,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:25:04,899 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:25:04,967 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:25:05,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:05,968 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:25:06,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:07,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:25:07,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:25:07,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:07,970 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:25:08,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:09,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:09,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:10,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:11,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:12,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:13,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:14,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:14,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:15,053 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:25:15,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:16,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:17,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:18,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:19,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:19,979 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:25:20,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:20,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:21,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:22,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:25:22,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:25:22,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:23,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:24,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:25,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:25,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:26,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:27,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:28,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:29,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:30,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:30,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:31,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:32,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:33,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:34,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:35,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:25:35,281 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:25:35,282 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:25:35,283 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:25:35,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:35,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:36,005 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:25:36,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:37,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:25:37,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:25:37,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:38,006 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:25:38,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:39,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:40,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:41,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:41,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:42,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:43,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:44,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:45,057 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:25:45,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:46,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:47,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:47,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:48,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:49,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:50,015 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:25:50,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:51,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:52,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:25:52,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:25:52,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:52,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:53,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:54,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:55,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:56,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:57,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:58,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:25:58,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:25:59,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:00,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:01,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:02,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:03,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:03,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:04,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:05,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:06,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:26:06,061 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:26:06,061 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:26:06,063 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:26:06,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:07,035 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:26:07,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:26:07,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:26:07,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:08,036 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:26:08,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:08,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:09,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:10,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:11,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:12,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:13,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:13,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:14,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:15,061 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:26:15,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:16,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:17,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:18,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:19,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:19,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:20,044 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:26:20,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:21,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:22,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:26:22,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:26:22,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:23,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:24,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:25,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:25,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:26,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:27,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:28,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:29,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:30,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:30,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:31,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:32,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:33,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:34,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:35,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:35,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:26:35,540 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:26:35,541 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:26:35,542 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:26:35,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:36,068 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:26:36,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:37,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:26:37,414 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:26:37,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:38,070 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:26:38,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:39,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:40,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:41,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:41,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:42,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:43,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:44,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:45,065 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:26:45,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:46,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:47,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:47,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:48,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:49,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:50,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:51,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:52,081 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:26:52,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:26:52,414 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:26:52,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:52,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:53,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:54,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:55,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:56,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:57,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:58,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:26:58,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:26:59,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:00,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:01,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:02,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:03,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:03,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:04,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:05,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:06,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:07,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:27:07,420 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:27:07,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:08,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:27:08,569 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:27:08,569 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:27:08,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:08,571 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:27:09,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:09,103 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:27:10,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:10,105 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:27:11,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:12,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:13,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:14,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:14,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:15,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:15,070 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:27:16,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:17,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:18,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:19,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:20,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:20,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:21,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:22,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:22,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:27:22,420 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:27:23,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:24,114 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:27:24,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:25,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:25,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:26,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:27,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:28,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:29,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:30,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:30,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:31,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:32,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:33,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:34,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:35,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:35,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:36,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:37,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:37,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:27:37,423 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:27:38,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:39,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:39,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:27:39,714 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:27:39,714 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:27:39,716 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:27:40,132 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:27:40,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:41,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:41,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:42,134 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:27:42,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:43,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:44,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:45,074 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:27:45,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:46,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:47,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:47,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:48,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:49,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:50,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:51,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:52,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:52,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:52,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:27:52,423 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:27:53,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:54,141 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:27:54,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:55,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:56,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:57,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:57,419 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:27:58,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:27:59,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:00,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:01,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:02,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:03,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:03,419 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:04,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:05,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:06,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:07,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:07,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:28:07,435 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:28:08,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:08,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:09,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:10,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:28:10,163 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:28:10,163 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:28:10,164 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:28:10,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:11,161 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:28:11,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:12,162 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:28:12,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:13,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:14,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:14,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:15,077 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:28:15,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:16,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:17,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:18,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:19,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:20,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:20,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:21,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:22,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:28:22,429 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:28:22,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:23,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:24,172 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:28:24,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:25,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:25,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:26,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:27,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:28,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:29,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:30,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:30,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:31,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:32,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:33,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:34,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:35,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:36,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:36,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:37,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:28:37,443 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:28:37,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:38,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:39,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:40,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:40,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:28:40,678 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:28:40,678 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:28:40,680 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:28:41,191 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:28:41,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:42,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:42,192 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:28:42,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:43,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:44,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:45,081 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:28:45,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:46,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:47,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:47,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:48,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:49,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:50,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:51,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:52,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:52,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:28:52,430 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:28:52,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:53,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:54,200 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:28:54,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:55,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:56,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:57,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:28:57,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:58,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:28:59,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:00,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:01,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:02,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:02,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:03,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:04,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:05,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:06,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:07,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:07,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:29:07,436 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:29:07,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:08,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:09,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:10,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:10,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:29:10,633 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:29:10,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:29:10,635 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:29:11,224 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:29:11,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:12,225 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:29:12,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:13,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:13,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:14,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:15,084 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:29:15,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:16,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:17,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:18,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:18,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:19,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:20,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:21,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:22,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:29:22,435 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:29:22,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:23,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:23,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:24,233 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:29:24,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:25,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:26,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:27,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:28,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:28,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:29,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:30,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:31,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:32,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:33,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:34,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:34,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:35,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:36,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:37,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:29:37,441 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:29:37,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:38,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:39,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:40,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:29:40,042 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:29:40,042 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:29:40,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:40,044 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:29:40,257 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:29:40,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:41,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:42,259 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:29:42,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:43,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:44,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:45,088 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:29:45,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:45,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:46,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:47,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:48,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:49,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:50,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:50,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:51,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:52,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:29:52,438 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:29:52,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:53,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:54,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:55,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:55,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:29:56,269 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:29:56,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:57,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:58,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:29:59,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:00,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:01,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:01,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:02,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:03,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:04,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:05,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:06,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:06,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:07,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:30:07,438 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:30:07,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:08,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:09,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:10,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:11,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:11,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:12,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:13,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:14,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:14,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:30:14,859 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:30:14,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:30:14,862 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:30:15,092 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:30:15,289 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:30:15,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:16,290 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:30:16,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:17,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:17,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:18,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:19,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:20,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:21,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:22,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:22,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:30:22,439 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:30:22,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:23,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:24,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:25,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:26,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:27,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:27,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:28,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:29,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:30,300 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:30:30,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:31,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:32,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:33,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:33,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:34,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:35,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:36,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:37,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:30:37,439 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:30:37,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:38,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:39,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:39,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:40,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:41,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:42,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:43,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:44,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:44,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:30:44,959 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:30:44,959 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:30:44,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:44,960 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:30:45,097 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:30:45,322 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:30:45,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:46,324 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:30:46,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:47,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:48,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:49,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:50,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:50,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:51,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:52,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:30:52,439 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:30:52,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:53,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:54,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:55,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:56,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:30:56,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:57,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:58,331 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:30:58,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:30:59,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:00,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:01,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:01,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:02,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:03,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:04,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:05,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:06,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:07,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:31:07,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:07,443 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:31:07,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:08,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:09,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:10,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:11,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:12,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:12,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:31:12,877 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:31:12,878 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:31:12,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:12,879 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:31:13,357 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:31:13,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:14,358 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:31:14,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:15,101 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:31:15,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:16,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:17,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:18,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:18,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:19,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:20,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:21,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:22,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:31:22,439 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:31:22,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:23,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:23,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:24,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:25,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:26,366 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:31:26,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:27,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:28,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:28,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:29,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:30,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:31,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:32,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:33,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:33,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:34,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:35,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:36,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:37,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:31:37,443 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:31:37,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:38,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:38,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:39,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:40,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:40,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:31:40,845 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:31:40,846 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:31:40,847 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:31:41,387 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:31:41,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:42,388 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:31:42,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:43,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:43,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:44,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:45,105 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:31:45,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:46,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:47,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:48,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:48,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:49,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:50,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:51,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:52,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:31:52,444 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:31:52,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:53,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:53,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:54,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:55,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:56,399 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:31:56,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:57,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:58,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:31:59,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:31:59,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:00,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:01,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:02,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:03,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:04,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:05,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:05,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:06,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:07,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:32:07,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:32:07,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:08,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:09,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:10,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:10,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:11,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:32:11,462 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:32:11,462 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:32:11,464 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:32:11,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:12,424 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:32:12,425 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:32:12,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:13,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:14,426 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:32:14,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:15,109 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:32:15,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:16,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:16,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:17,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:18,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:19,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:20,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:21,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:21,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:22,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:32:22,447 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:32:22,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:23,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:24,433 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:32:24,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:25,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:26,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:26,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:27,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:28,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:29,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:30,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:31,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:32,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:32,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:33,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:34,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:35,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:36,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:37,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:32:37,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:32:37,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:37,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:38,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:39,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:40,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:32:40,804 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:32:40,805 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:32:40,807 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:32:41,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:41,453 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:32:42,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:42,454 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:32:43,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:43,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:44,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:45,113 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:32:45,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:46,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:47,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:48,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:48,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:49,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:50,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:51,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:52,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:52,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:32:52,450 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:32:53,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:53,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:32:54,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:55,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:56,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:56,464 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:32:57,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:58,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:59,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:32:59,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:00,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:01,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:02,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:03,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:04,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:05,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:05,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:06,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:07,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:07,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:33:07,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:33:08,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:09,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:10,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:10,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:11,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:12,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:33:12,117 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:33:12,118 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:33:12,119 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:33:12,487 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:33:12,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:13,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:14,489 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:33:14,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:15,117 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:33:15,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:16,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:16,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:17,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:18,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:19,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:20,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:21,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:21,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:22,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:33:22,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:33:22,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:23,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:24,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:25,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:26,498 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:33:26,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:27,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:27,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:28,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:29,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:30,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:31,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:32,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:32,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:33,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:34,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:35,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:36,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:37,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:33:37,469 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:33:37,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:37,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:38,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:39,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:40,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:41,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:42,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:42,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:33:42,554 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:33:42,554 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:33:42,556 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:33:42,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:43,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:43,524 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:33:44,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:44,525 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:33:45,121 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:33:45,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:46,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:47,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:48,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:48,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:49,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:50,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:51,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:52,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:33:52,453 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:33:52,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:53,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:53,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:54,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:55,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:56,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:57,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:58,537 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:33:58,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:33:58,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:33:59,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:00,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:01,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:02,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:03,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:04,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:04,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:05,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:06,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:07,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:34:07,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:34:07,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:08,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:09,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:10,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:10,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:11,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:12,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:13,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:14,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:15,128 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:34:15,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:16,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:16,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:17,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:34:17,408 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:34:17,409 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:34:17,411 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:34:17,562 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:34:17,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:18,563 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:34:18,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:19,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:20,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:21,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:21,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:22,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:34:22,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:34:22,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:23,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:24,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:25,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:26,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:27,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:27,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:28,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:29,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:30,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:31,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:32,573 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:34:32,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:32,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:33,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:34,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:35,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:36,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:37,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:34:37,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:34:37,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:37,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:38,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:39,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:40,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:41,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:42,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:43,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:43,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:44,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:45,133 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:34:45,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:46,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:47,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:48,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:34:48,687 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:34:48,688 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:34:48,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:48,689 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:34:49,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:49,593 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:34:49,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:50,594 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:34:50,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:51,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:52,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:34:52,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:34:52,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:53,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:54,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:34:54,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:55,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:56,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:57,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:58,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:34:59,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:00,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:00,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:01,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:02,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:03,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:04,604 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:35:04,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:05,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:05,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:06,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:07,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:35:07,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:35:07,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:08,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:09,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:10,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:11,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:11,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:12,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:13,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:14,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:15,136 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:35:15,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:16,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:17,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:17,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:18,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:19,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:20,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:21,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:35:21,483 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:35:21,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:35:21,486 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:35:21,623 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:35:21,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:22,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:35:22,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:35:22,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:22,624 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:35:22,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:23,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:24,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:25,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:26,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:27,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:27,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:28,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:29,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:30,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:31,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:32,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:33,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:33,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:34,633 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:35:34,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:35,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:36,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:37,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:35:37,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:35:37,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:38,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:38,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:39,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:40,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:41,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:42,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:43,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:44,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:44,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:45,148 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:35:45,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:46,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:47,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:48,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:49,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:49,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:50,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:35:50,324 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:35:50,324 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:35:50,325 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:35:50,655 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:35:50,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:51,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:52,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:35:52,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:35:52,657 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:35:52,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:53,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:54,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:54,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:55,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:56,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:57,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:58,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:35:59,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:35:59,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:00,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:01,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:02,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:03,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:04,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:04,666 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:36:04,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:05,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:06,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:07,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:36:07,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:36:07,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:08,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:09,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:10,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:10,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:11,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:12,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:13,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:14,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:15,147 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:36:15,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:16,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:16,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:17,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:18,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:19,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:20,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:36:20,178 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:36:20,179 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:36:20,181 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:36:20,696 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:36:20,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:21,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:21,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:22,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:36:22,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:36:22,697 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:36:22,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:23,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:24,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:25,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:26,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:26,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:27,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:28,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:29,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:30,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:31,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:31,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:32,704 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:36:32,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:33,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:34,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:35,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:36,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:37,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:36:37,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:36:37,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:37,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:38,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:39,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:40,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:41,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:42,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:42,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:43,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:44,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:45,148 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:36:45,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:46,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:47,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:48,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:48,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:48,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:36:48,966 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:36:48,967 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:36:48,968 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:36:49,723 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:36:49,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:50,724 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:36:50,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:51,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:52,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:36:52,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:36:52,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:53,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:53,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:54,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:55,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:56,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:57,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:58,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:36:58,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:36:59,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:00,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:01,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:02,733 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:37:02,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:03,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:03,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:04,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:05,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:06,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:07,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:37:07,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:37:07,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:08,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:09,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:09,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:10,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:11,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:12,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:13,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:14,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:14,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:15,154 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:37:15,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:16,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:37:16,364 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:37:16,364 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:37:16,366 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:37:16,752 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:37:16,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:17,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:18,754 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:37:18,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:19,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:19,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:20,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:21,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:22,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:37:22,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:37:22,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:23,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:24,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:24,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:25,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:26,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:27,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:28,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:29,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:30,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:30,762 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:37:30,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:31,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:32,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:33,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:34,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:35,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:35,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:36,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:37,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:37:37,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:37:37,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:38,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:39,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:40,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:40,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:41,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:42,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:43,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:44,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:45,156 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:37:45,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:37:45,557 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:37:45,557 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:37:45,558 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:37:45,789 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:37:45,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:45,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:46,790 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:37:46,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:47,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:48,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:49,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:50,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:51,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:51,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:52,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:37:52,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:37:52,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:53,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:54,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:55,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:56,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:57,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:37:57,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:58,798 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:37:58,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:37:59,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:00,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:01,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:02,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:02,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:03,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:04,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:05,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:06,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:07,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:38:07,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:38:07,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:08,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:08,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:09,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:10,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:11,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:12,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:13,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:13,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:14,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:38:14,160 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:38:14,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:38:14,166 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:38:14,827 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:38:14,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:15,159 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:38:15,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:16,828 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:38:16,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:17,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:18,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:18,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:19,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:20,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:21,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:22,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:38:22,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:38:22,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:23,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:24,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:24,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:25,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:27,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:28,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:28,837 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:38:29,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:29,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:30,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:31,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:32,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:33,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:34,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:35,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:35,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:36,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:37,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:37,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:38:37,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:38:38,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:39,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:40,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:40,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:41,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:42,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:43,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:44,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:44,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:38:44,284 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:38:44,284 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:38:44,289 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:38:44,870 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:38:45,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:45,164 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:38:46,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:46,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:46,872 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:38:47,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:48,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:49,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:50,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:51,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:51,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:52,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:52,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:38:52,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:38:53,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:54,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:55,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:56,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:56,880 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:38:56,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:38:57,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:58,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:38:59,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:00,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:01,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:02,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:02,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:03,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:04,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:05,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:06,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:07,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:07,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:39:07,472 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:39:08,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:08,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:09,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:10,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:11,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:12,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:13,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:13,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:39:13,162 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:39:13,162 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:39:13,163 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:39:13,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:13,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:39:14,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:14,908 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:39:15,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:15,168 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:39:16,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:17,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:18,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:19,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:19,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:20,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:21,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:22,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:22,458 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:39:22,458 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:39:23,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:24,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:25,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:25,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:26,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:26,917 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:39:27,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:28,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:29,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:30,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:30,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:31,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:32,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:33,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:34,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:35,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:36,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:36,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:37,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:37,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:39:37,474 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:39:38,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:39,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:40,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:41,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:41,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:42,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:42,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:39:42,974 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:39:42,974 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:39:42,975 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:39:43,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:43,939 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:39:44,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:44,940 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:39:45,172 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:39:45,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:46,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:47,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:47,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:48,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:49,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:50,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:51,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:52,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:52,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:39:52,463 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:39:52,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:53,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:54,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:55,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:56,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:56,950 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:39:57,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:58,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:39:58,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:39:59,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:00,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:01,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:02,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:03,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:03,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:04,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:05,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:06,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:07,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:07,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:40:07,463 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:40:08,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:08,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:09,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:10,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:11,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:12,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:40:12,356 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:40:12,356 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:40:12,357 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:40:12,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:12,973 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:40:13,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:13,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:14,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:14,974 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:40:15,175 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:40:15,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:16,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:17,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:18,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:19,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:19,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:20,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:21,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:22,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:40:22,463 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:40:22,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:23,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:24,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:25,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:25,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:26,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:27,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:28,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:28,984 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:40:29,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:30,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:30,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:31,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:32,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:33,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:34,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:35,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:36,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:36,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:37,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:40:37,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:40:37,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:38,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:39,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:40,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:41,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:41,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:42,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:43,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:44,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:45,179 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:40:45,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:45,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:40:45,819 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:40:45,819 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:40:45,821 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:40:46,001 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:40:46,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:47,003 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:40:47,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:47,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:48,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:49,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:50,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:51,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:52,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:40:52,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:40:52,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:52,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:53,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:54,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:55,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:56,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:57,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:58,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:40:58,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:40:59,011 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:40:59,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:00,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:01,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:02,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:03,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:03,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:04,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:05,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:06,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:07,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:41:07,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:41:07,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:08,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:08,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:09,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:10,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:11,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:12,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:13,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:13,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:14,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:41:14,480 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:41:14,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:41:14,481 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:41:14,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:15,033 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:41:15,184 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:41:15,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:16,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:17,035 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:41:17,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:18,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:18,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:19,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:20,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:21,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:22,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:41:22,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:41:22,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:23,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:24,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:24,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:25,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:26,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:27,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:28,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:29,043 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:41:29,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:29,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:30,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:31,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:32,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:33,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:34,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:35,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:35,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:36,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:37,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:41:37,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:41:37,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:38,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:39,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:40,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:40,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:41,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:42,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:43,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:43,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:41:43,995 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:41:43,996 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:41:43,997 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:41:44,058 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:41:44,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:45,060 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:41:45,188 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:41:45,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:46,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:46,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:47,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:48,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:49,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:50,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:51,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:51,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:52,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:41:52,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:41:52,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:53,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:54,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:55,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:56,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:56,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:41:57,068 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:41:57,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:58,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:41:59,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:00,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:01,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:02,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:02,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:03,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:04,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:05,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:06,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:07,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:42:07,465 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:42:07,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:08,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:08,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:09,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:10,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:11,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:12,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:13,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:42:13,313 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:42:13,313 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:42:13,314 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:42:13,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:13,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:14,085 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:42:14,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:15,086 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:42:15,192 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:42:15,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:16,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:17,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:18,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:19,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:19,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:20,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:21,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:22,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:42:22,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:42:22,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:23,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:24,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:25,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:25,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:26,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:27,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:28,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:29,096 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:42:29,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:30,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:30,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:31,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:32,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:33,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:34,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:35,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:36,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:36,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:37,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:42:37,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:42:37,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:38,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:39,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:40,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:41,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:41,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:42,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:43,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:44,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:42:44,646 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:42:44,647 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:42:44,648 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:42:45,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:45,127 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:42:45,196 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:42:45,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:46,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:47,129 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:42:47,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:47,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:48,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:49,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:50,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:51,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:52,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:42:52,465 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:42:52,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:52,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:53,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:54,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:55,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:56,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:57,137 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:42:57,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:58,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:42:58,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:42:59,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:00,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:01,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:02,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:03,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:03,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:04,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:05,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:06,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:07,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:43:07,489 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:43:07,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:08,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:08,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:09,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:10,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:11,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:12,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:13,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:43:13,182 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:43:13,182 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:43:13,184 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:43:13,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:13,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:14,156 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:43:14,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:15,157 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:43:15,200 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:43:15,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:16,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:17,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:18,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:18,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:19,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:20,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:21,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:22,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:43:22,475 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:43:22,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:23,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:24,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:24,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:25,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:26,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:27,166 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:43:27,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:28,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:29,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:29,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:30,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:31,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:32,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:33,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:34,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:35,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:35,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:36,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:37,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:43:37,485 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:43:37,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:38,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:39,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:40,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:40,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:41,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:41,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:43:41,888 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:43:41,888 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:43:41,889 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:43:42,187 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:43:42,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:43,188 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:43:43,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:44,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:45,204 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:43:45,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:46,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:46,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:47,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:48,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:49,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:50,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:51,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:51,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:52,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:43:52,479 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:43:52,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:53,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:54,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:55,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:56,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:57,196 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:43:57,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:43:57,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:58,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:43:59,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:00,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:01,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:02,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:02,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:03,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:04,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:05,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:06,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:07,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:44:07,479 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:44:07,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:08,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:08,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:09,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:10,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:11,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:12,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:13,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:13,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:14,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:15,208 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:44:15,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:44:15,271 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:44:15,272 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:44:15,273 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:44:15,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:16,215 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:44:16,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:17,216 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:44:17,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:18,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:18,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:19,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:20,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:21,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:22,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:44:22,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:44:22,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:23,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:24,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:24,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:25,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:26,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:27,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:28,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:29,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:30,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:30,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:31,228 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:44:31,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:32,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:33,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:34,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:35,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:35,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:36,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:37,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:44:37,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:44:37,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:38,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:39,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:40,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:40,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:41,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:42,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:43,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:44,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:45,212 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:44:45,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:46,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:46,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:44:46,348 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:44:46,348 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:44:46,349 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:44:46,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:47,250 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:44:47,250 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:44:47,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:48,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:49,251 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:44:49,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:50,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:51,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:51,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:52,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:44:52,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:44:52,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:53,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:54,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:55,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:56,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:57,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:44:57,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:59,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:44:59,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:00,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:01,260 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:45:01,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:02,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:03,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:03,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:04,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:05,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:06,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:07,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:45:07,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:45:07,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:08,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:08,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:09,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:10,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:11,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:12,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:13,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:13,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:14,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:15,215 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:45:15,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:16,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:45:16,416 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:45:16,416 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:45:16,417 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:45:16,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:17,279 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:45:17,280 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:45:17,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:18,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:19,281 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:45:19,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:19,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:20,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:21,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:22,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:45:22,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:45:22,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:23,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:24,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:25,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:25,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:26,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:27,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:28,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:29,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:30,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:30,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:31,290 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:45:31,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:32,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:33,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:34,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:35,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:36,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:36,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:37,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:45:37,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:45:37,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:38,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:39,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:40,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:41,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:41,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:42,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:43,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:44,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:45,222 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:45:45,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:46,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:45:46,262 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:45:46,263 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:45:46,264 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:45:46,309 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:45:46,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:47,310 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:45:47,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:47,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:48,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:49,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:50,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:51,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:52,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:45:52,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:45:52,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:53,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:53,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:54,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:55,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:56,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:57,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:58,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:45:59,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:45:59,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:00,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:01,319 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:46:01,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:02,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:03,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:03,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:04,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:05,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:06,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:07,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:46:07,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:46:07,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:08,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:09,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:09,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:10,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:11,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:12,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:13,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:14,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:14,978 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:15,233 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:46:15,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:16,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:46:16,517 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:46:16,518 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:46:16,519 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:46:16,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:17,343 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:46:17,343 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:46:17,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:18,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:19,345 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:46:19,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:20,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:20,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:21,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:22,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:46:22,481 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:46:22,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:23,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:24,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:25,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:26,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:26,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:27,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:29,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:30,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:31,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:31,354 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:46:32,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:32,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:33,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:34,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:35,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:36,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:37,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:37,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:46:37,480 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:46:37,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:38,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:39,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:40,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:41,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:42,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:43,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:43,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:44,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:45,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:46:45,190 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:46:45,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:46:45,192 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:46:45,230 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:46:45,375 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:46:45,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:46,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:47,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:47,377 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:46:48,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:48,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:49,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:50,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:51,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:52,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:52,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:46:52,481 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:46:53,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:54,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:54,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:55,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:56,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:57,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:58,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:46:59,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:46:59,385 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:46:59,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:00,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:01,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:02,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:03,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:04,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:05,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:05,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:06,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:07,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:47:07,481 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:47:07,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:08,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:09,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:10,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:10,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:11,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:12,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:13,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:14,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:15,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:15,234 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:47:15,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:47:15,289 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:47:15,289 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:47:15,291 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:47:15,413 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:47:15,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:16,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:17,414 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:47:17,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:18,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:19,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:20,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:21,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:21,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:22,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:47:22,481 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:47:22,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:23,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:24,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:25,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:26,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:26,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:27,421 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:47:27,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:28,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:29,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:30,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:31,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:32,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:32,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:33,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:34,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:35,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:36,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:37,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:37,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:47:37,481 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:47:37,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:38,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:39,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:40,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:41,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:42,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:42,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:43,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:47:43,624 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:47:43,624 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:47:43,625 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:47:44,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:44,444 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:47:45,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:45,241 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:47:45,446 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:47:46,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:47,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:47,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:48,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:49,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:50,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:51,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:52,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:52,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:47:52,481 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:47:52,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:53,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:54,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:55,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:56,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:57,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:58,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:58,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:47:59,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:47:59,455 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:48:00,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:01,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:02,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:03,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:03,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:04,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:05,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:06,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:07,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:07,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:48:07,481 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:48:08,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:08,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:09,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:10,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:11,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:12,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:13,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:14,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:14,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:15,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:15,246 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:48:15,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:48:15,738 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:48:15,739 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:48:15,740 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:48:16,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:16,474 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:48:17,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:17,476 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:48:18,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:19,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:19,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:20,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:21,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:22,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:22,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:48:22,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:48:23,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:24,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:25,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:25,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:26,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:27,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:28,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:29,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:30,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:30,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:31,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:31,485 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:48:32,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:33,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:34,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:35,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:36,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:36,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:37,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:37,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:48:37,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:48:38,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:39,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:40,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:41,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:41,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:42,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:43,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:44,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:45,250 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:48:45,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:46,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:46,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:48:46,849 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:48:46,850 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:48:46,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:46,851 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:48:47,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:47,511 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:48:47,512 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:48:48,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:49,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:49,513 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:48:50,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:51,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:52,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:52,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:52,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:48:52,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:48:53,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:54,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:55,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:56,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:57,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:58,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:48:58,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:48:59,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:00,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:01,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:02,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:03,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:03,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:03,522 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:49:04,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:05,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:06,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:07,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:07,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:49:07,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:49:08,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:08,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:09,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:10,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:11,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:12,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:13,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:14,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:14,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:15,254 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:49:15,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:16,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:17,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:18,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:18,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:49:18,955 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:49:18,956 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:49:18,957 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:49:19,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:19,541 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:49:19,541 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:49:20,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:20,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:21,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:21,544 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:49:22,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:49:22,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:49:22,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:23,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:24,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:25,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:26,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:26,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:27,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:28,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:29,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:30,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:31,552 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:49:31,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:31,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:32,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:33,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:34,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:35,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:36,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:37,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:37,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:49:37,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:49:37,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:38,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:39,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:40,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:41,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:42,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:42,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:43,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:44,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:45,253 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:49:45,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:45,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:49:45,878 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:49:45,879 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:49:45,880 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:49:46,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:46,578 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:49:47,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:47,581 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:49:48,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:48,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:49,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:50,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:51,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:52,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:49:52,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:49:52,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:53,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:53,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:54,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:55,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:56,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:57,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:58,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:59,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:49:59,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:49:59,588 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:50:00,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:01,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:02,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:03,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:04,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:04,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:05,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:06,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:07,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:50:07,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:50:07,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:08,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:09,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:10,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:10,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:11,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:12,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:13,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:14,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:15,255 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:50:15,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:16,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:16,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:50:16,349 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:50:16,349 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:50:16,350 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:50:16,609 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:50:16,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:17,610 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:50:17,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:18,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:19,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:20,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:21,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:21,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:22,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:50:22,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:50:22,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:23,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:24,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:25,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:26,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:26,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:27,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:28,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:29,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:30,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:31,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:31,621 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:50:31,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:32,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:33,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:34,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:35,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:36,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:37,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:50:37,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:50:37,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:37,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:38,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:39,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:40,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:41,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:42,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:43,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:43,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:44,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:45,270 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:50:45,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:46,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:46,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:50:46,904 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:50:46,904 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:50:46,906 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:50:47,646 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:50:47,647 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:50:47,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:48,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:49,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:49,648 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:50:49,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:50,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:51,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:52,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:50:52,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:50:52,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:53,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:54,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:54,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:55,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:56,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:57,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:58,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:50:59,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:50:59,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:00,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:01,664 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:51:01,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:02,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:03,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:04,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:05,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:05,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:06,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:07,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:51:07,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:51:07,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:08,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:09,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:10,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:11,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:11,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:12,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:13,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:14,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:15,272 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:51:15,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:16,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:51:16,678 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:51:16,678 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:51:16,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:16,683 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:51:16,684 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:51:17,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:17,686 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:51:18,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:19,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:20,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:21,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:22,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:22,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:51:22,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:51:22,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:23,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:24,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:25,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:26,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:27,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:28,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:28,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:29,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:29,694 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:51:30,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:31,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:32,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:33,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:34,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:34,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:35,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:36,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:37,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:37,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:51:37,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:51:38,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:39,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:39,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:40,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:41,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:42,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:43,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:44,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:44,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:45,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:45,276 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:51:46,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:46,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:51:46,457 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:51:46,457 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:51:46,459 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:51:46,713 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:51:47,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:47,714 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:51:48,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:49,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:50,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:50,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:51,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:52,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:52,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:51:52,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:51:53,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:54,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:55,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:56,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:56,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:51:57,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:58,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:51:59,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:00,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:01,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:01,722 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:52:02,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:02,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:03,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:04,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:05,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:06,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:07,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:07,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:52:07,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:52:07,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:08,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:09,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:10,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:11,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:12,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:13,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:13,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:14,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:15,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:15,279 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:52:16,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:17,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:17,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:52:17,662 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:52:17,662 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:52:17,667 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:52:17,746 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:52:18,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:19,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:19,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:19,748 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:52:20,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:21,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:22,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:22,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:52:22,483 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:52:23,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:24,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:24,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:25,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:26,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:27,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:28,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:29,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:30,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:30,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:31,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:31,756 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:52:32,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:33,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:34,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:35,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:35,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:36,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:37,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:37,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:52:37,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:52:38,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:39,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:40,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:41,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:41,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:42,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:43,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:44,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:45,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:45,283 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:52:46,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:47,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:47,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:47,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:52:47,597 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:52:47,598 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:52:47,599 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:52:47,775 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:52:48,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:49,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:49,777 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:52:50,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:51,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:52,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:52,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:52:52,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:52:52,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:53,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:54,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:55,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:56,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:57,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:58,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:52:58,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:52:59,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:00,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:01,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:02,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:03,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:03,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:03,786 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:53:04,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:05,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:06,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:07,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:07,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:53:07,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:53:08,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:09,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:09,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:10,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:11,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:12,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:13,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:14,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:14,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:15,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:15,293 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:53:16,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:17,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:18,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:19,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:19,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:53:19,771 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:53:19,772 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:53:19,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:19,778 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:53:19,807 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:53:20,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:21,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:21,809 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:53:22,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:22,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:53:22,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:53:23,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:24,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:25,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:25,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:26,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:27,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:28,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:29,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:30,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:30,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:31,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:32,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:33,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:33,819 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:53:34,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:35,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:36,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:36,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:37,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:37,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:53:37,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:53:38,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:39,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:40,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:41,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:41,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:42,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:43,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:44,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:45,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:45,286 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:53:46,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:47,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:47,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:48,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:49,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:50,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:50,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:53:50,649 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:53:50,650 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:53:50,655 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:53:50,839 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:53:51,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:51,840 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:53:52,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:52,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:53:52,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:53:52,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:53,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:54,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:55,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:56,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:57,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:58,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:53:58,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:53:59,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:00,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:01,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:02,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:03,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:04,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:04,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:05,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:05,852 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:54:06,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:07,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:07,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:54:07,484 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:54:08,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:09,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:09,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:10,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:11,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:12,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:13,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:14,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:15,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:15,290 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:54:15,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:16,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:17,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:18,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:19,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:20,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:20,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:21,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:22,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:22,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:54:22,490 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:54:23,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:23,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:54:23,545 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:54:23,545 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:54:23,548 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:54:23,871 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:54:24,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:25,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:25,873 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:54:26,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:26,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:27,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:28,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:29,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:30,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:31,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:31,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:32,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:33,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:34,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:35,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:35,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:54:36,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:37,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:37,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:37,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:54:37,487 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:54:38,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:39,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:40,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:41,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:42,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:42,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:43,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:44,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:45,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:45,293 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:54:46,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:47,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:48,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:48,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:49,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:50,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:50,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:54:50,267 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:54:50,267 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:54:50,268 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:54:50,907 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:54:51,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:51,909 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:54:52,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:52,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:54:52,487 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:54:53,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:53,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:54:54,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:55,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:56,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:57,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:58,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:59,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:54:59,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:00,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:01,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:02,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:03,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:03,917 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:55:04,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:04,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:05,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:06,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:07,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:07,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:55:07,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:55:08,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:09,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:10,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:10,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:11,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:12,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:13,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:14,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:15,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:15,296 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:55:16,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:16,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:17,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:18,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:55:18,190 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:55:18,190 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:55:18,191 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:55:18,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:18,935 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:55:19,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:19,936 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:55:20,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:21,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:21,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:22,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:55:22,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:55:22,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:23,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:24,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:25,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:26,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:26,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:27,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:28,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:29,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:30,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:31,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:31,946 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:55:31,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:32,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:33,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:34,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:35,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:36,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:37,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:55:37,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:55:37,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:37,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:38,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:39,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:40,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:41,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:42,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:43,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:43,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:44,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:45,303 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:55:45,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:46,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:47,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:48,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:55:48,279 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:55:48,280 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:55:48,282 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:55:48,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:48,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:48,967 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:55:49,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:49,968 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:55:50,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:51,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:52,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:55:52,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:55:52,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:53,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:54,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:54,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:55,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:56,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:57,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:58,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:55:59,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:55:59,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:00,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:01,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:02,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:03,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:03,978 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:56:04,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:04,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:05,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:06,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:07,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:56:07,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:56:07,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:08,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:09,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:09,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:10,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:11,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:12,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:13,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:14,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:14,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:15,300 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:56:15,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:16,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:17,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:18,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:19,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:19,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:56:19,580 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:56:19,581 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:56:19,586 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:56:19,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:19,999 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:56:19,999 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:56:20,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:21,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:22,000 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:56:22,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:56:22,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:56:22,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:23,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:24,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:24,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:25,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:26,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:27,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:28,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:29,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:29,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:30,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:31,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:32,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:33,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:34,009 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:56:34,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:34,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:35,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:36,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:37,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:56:37,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:56:37,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:38,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:39,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:40,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:40,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:41,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:42,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:43,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:45,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:45,310 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:56:45,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:46,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:46,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:47,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:48,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:49,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:56:49,391 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:56:49,391 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:56:49,393 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:56:49,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:50,034 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:56:50,034 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:56:50,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:51,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:51,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:52,035 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:56:52,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:56:52,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:56:52,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:53,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:54,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:55,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:56,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:57,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:56:57,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:58,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:56:59,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:00,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:02,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:03,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:03,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:04,044 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:57:04,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:05,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:06,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:07,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:07,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:57:07,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:57:08,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:08,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:09,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:10,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:11,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:12,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:13,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:13,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:14,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:15,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:15,316 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:57:16,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:17,118 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:18,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:19,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:19,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:19,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:57:19,384 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:57:19,385 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:57:19,387 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:57:20,058 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:57:20,059 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:57:20,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:21,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:22,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:22,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:57:22,488 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:57:23,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:24,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:24,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:25,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:26,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:27,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:28,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:29,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:29,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:30,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:31,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:32,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:33,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:34,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:34,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:35,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:36,070 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:57:36,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:37,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:37,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:57:37,489 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:57:38,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:39,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:40,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:40,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:41,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:42,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:43,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:44,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:45,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:45,313 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:57:46,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:46,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:47,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:48,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:49,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:50,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:51,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:51,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:52,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:52,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:57:52,489 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:57:52,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:57:52,760 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:57:52,760 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:57:52,766 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:57:53,100 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:57:53,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:54,101 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:57:54,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:55,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:56,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:57,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:57,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:57:58,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:57:59,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:00,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:01,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:02,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:03,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:03,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:04,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:05,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:06,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:07,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:07,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:58:07,489 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:58:08,111 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:58:08,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:09,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:09,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:10,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:11,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:12,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:13,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:14,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:14,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:15,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:15,318 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:58:16,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:17,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:18,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:19,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:19,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:20,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:21,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:22,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:22,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:58:22,501 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:58:23,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:23,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:58:23,550 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:58:23,551 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:58:23,553 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:58:24,135 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:58:24,136 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:58:24,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:25,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:25,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:26,137 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:58:26,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:27,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:28,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:29,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:30,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:31,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:31,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:32,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:33,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:34,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:35,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:36,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:37,147 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:58:37,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:37,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:37,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:58:37,494 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:58:38,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:39,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:40,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:41,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:42,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:42,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:43,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:44,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:45,321 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:58:45,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:46,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:47,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:48,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:48,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:49,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:50,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:51,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:52,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:52,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:58:52,495 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:58:52,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:58:52,608 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:58:52,608 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:58:52,610 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:58:53,170 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:58:53,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:54,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:58:54,176 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:58:54,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:55,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:56,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:57,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:58,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:59,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:58:59,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:00,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:01,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:02,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:03,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:04,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:04,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:05,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:06,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:07,184 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:59:07,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:07,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:59:07,495 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:59:08,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:09,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:10,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:10,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:11,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:12,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:13,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:14,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:15,326 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:59:15,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:16,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:16,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:17,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:18,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:19,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:20,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:21,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:21,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:22,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:59:22,220 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:59:22,220 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:59:22,221 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:59:22,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:22,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:59:22,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:59:23,201 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:59:23,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:24,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:25,203 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:59:25,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:26,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:27,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:27,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:28,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:29,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:30,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:31,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:32,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:33,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:33,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:34,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:35,210 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:59:35,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:36,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:37,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:59:37,621 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:59:37,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:38,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:39,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:39,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:40,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:41,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:42,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:43,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:44,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:44,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:45,330 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 08:59:45,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:46,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:47,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:48,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:49,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:49,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:50,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:51,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 08:59:51,158 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 08:59:51,158 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 08:59:51,160 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 08:59:51,228 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 08:59:51,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:52,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 08:59:52,621 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 08:59:52,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:53,230 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 08:59:53,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:54,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:55,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:55,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 08:59:56,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:57,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:58,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 08:59:59,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:00,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:01,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:01,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:02,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:03,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:04,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:05,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:06,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:07,240 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:00:07,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:07,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:00:07,621 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:00:07,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:08,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:09,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:10,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:11,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:12,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:12,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:13,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:14,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:15,334 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:00:15,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:16,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:17,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:18,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:18,692 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:19,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:20,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:21,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:22,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:00:22,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:00:22,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:23,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:23,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:24,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:00:24,346 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:00:24,347 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:00:24,348 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:00:24,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:25,266 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:00:25,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:26,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:27,268 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:00:27,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:28,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:28,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:29,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:30,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:31,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:32,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:33,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:34,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:34,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:35,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:36,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:37,275 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:00:37,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:00:37,626 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:00:37,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:38,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:39,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:39,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:40,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:41,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:42,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:43,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:44,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:45,339 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:00:45,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:45,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:46,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:47,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:48,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:49,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:50,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:50,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:51,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:52,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:00:52,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:00:52,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:52,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:00:52,872 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:00:52,872 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:00:52,874 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:00:53,301 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:00:53,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:54,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:55,302 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:00:55,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:56,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:00:56,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:57,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:58,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:00:59,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:00,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:01,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:02,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:02,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:03,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:04,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:05,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:06,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:07,310 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:01:07,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:01:07,626 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:01:07,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:08,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:08,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:09,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:10,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:11,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:12,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:13,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:13,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:14,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:15,342 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:01:15,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:16,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:17,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:18,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:19,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:19,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:20,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:21,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:01:21,750 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:01:21,750 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:01:21,752 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:01:22,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:22,329 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:01:22,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:01:22,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:01:22,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:23,330 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:01:23,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:24,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:24,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:25,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:26,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:27,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:28,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:29,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:29,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:30,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:31,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:32,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:33,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:34,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:34,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:35,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:36,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:37,341 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:01:37,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:01:37,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:01:37,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:38,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:39,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:39,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:40,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:41,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:42,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:43,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:44,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:44,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:45,353 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:01:45,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:46,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:47,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:48,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:49,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:49,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:50,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:01:50,897 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:01:50,897 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:01:50,899 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:01:50,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:51,364 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:01:51,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:52,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:01:52,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:01:52,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:53,366 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:01:53,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:54,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:55,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:01:55,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:56,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:57,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:58,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:01:59,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:00,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:01,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:01,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:02,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:03,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:04,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:05,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:06,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:07,374 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:02:07,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:02:07,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:02:07,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:07,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:08,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:09,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:10,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:11,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:12,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:12,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:13,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:14,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:15,350 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:02:15,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:16,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:17,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:18,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:18,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:19,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:20,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:21,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:22,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:02:22,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:02:22,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:02:22,893 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:02:22,894 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:02:22,895 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:02:23,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:23,393 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:02:24,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:24,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:25,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:25,395 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:02:26,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:27,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:28,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:29,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:29,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:30,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:31,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:32,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:33,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:34,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:34,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:35,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:36,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:37,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:37,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:02:37,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:02:38,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:39,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:39,405 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:02:39,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:40,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:41,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:42,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:43,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:44,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:45,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:45,354 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:02:45,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:46,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:47,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:48,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:49,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:50,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:50,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:51,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:52,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:52,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:02:52,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:02:53,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:54,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:55,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:55,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:02:56,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:57,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:58,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:58,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:02:58,241 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:02:58,242 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:02:58,244 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:02:58,427 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:02:59,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:02:59,428 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:03:00,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:00,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:01,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:01,429 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:03:02,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:03,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:04,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:05,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:05,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:06,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:07,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:07,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:03:07,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:03:08,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:09,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:10,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:11,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:11,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:12,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:13,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:14,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:15,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:15,359 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:03:15,438 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:03:16,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:17,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:17,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:18,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:19,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:20,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:21,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:22,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:22,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:03:22,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:03:22,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:23,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:24,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:25,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:26,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:27,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:27,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:28,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:29,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:29,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:03:29,871 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:03:29,871 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:03:29,873 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:03:30,461 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:03:30,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:31,462 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:03:31,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:32,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:33,464 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:03:33,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:33,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:34,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:35,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:36,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:37,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:37,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:03:37,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:03:38,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:38,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:39,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:40,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:41,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:42,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:43,470 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:03:43,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:43,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:44,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:45,363 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:03:45,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:46,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:47,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:48,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:49,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:49,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:50,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:51,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:52,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:52,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:03:52,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:03:53,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:54,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:54,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:03:55,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:56,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:57,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:58,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:03:59,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:03:59,416 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:03:59,417 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:03:59,418 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:03:59,493 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:04:00,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:00,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:00,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:01,496 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:04:01,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:02,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:03,497 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:04:03,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:04,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:05,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:06,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:06,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:07,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:07,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:04:07,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:04:08,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:09,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:10,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:11,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:12,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:12,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:13,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:14,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:15,367 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:04:15,507 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:04:15,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:16,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:17,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:17,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:18,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:19,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:20,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:21,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:22,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:22,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:22,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:04:22,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:04:23,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:24,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:25,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:26,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:27,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:27,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:28,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:29,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:30,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:04:30,046 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:04:30,047 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:04:30,049 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:04:30,525 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:04:30,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:31,526 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:04:31,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:32,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:32,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:33,528 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:04:33,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:34,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:35,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:36,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:37,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:37,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:04:37,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:04:37,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:38,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:39,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:40,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:41,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:42,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:43,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:43,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:44,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:45,371 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:04:45,536 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:04:45,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:46,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:47,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:48,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:49,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:49,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:50,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:51,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:52,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:04:52,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:04:52,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:53,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:54,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:54,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:04:55,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:56,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:57,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:58,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:04:59,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:00,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:00,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:00,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:05:00,757 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:05:00,757 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:05:00,758 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:05:01,559 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:05:01,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:02,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:03,561 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:05:03,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:04,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:05,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:05,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:06,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:07,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:05:07,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:05:07,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:08,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:09,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:10,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:11,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:11,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:12,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:13,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:14,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:15,375 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:05:15,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:16,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:16,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:17,572 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:05:17,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:18,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:19,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:20,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:21,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:21,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:22,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:05:22,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:05:22,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:23,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:24,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:25,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:26,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:27,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:27,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:28,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:29,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:30,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:31,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:32,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:32,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:33,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:34,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:05:34,222 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:05:34,222 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:05:34,224 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:05:34,602 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:05:34,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:35,603 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:05:35,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:36,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:37,605 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:05:37,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:05:37,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:05:37,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:37,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:38,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:39,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:40,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:41,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:42,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:43,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:43,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:44,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:45,379 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:05:45,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:46,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:47,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:48,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:49,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:49,614 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:05:49,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:50,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:51,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:52,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:05:52,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:05:52,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:53,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:54,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:54,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:05:55,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:56,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:57,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:58,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:05:59,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:00,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:00,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:01,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:02,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:03,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:04,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:06:04,545 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:06:04,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:06:04,547 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:06:04,636 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:06:04,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:05,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:06,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:06,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:07,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:06:07,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:06:07,639 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:06:07,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:08,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:09,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:10,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:11,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:12,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:12,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:13,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:14,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:15,383 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:06:15,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:16,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:17,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:17,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:18,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:19,649 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:06:19,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:20,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:21,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:22,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:06:22,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:06:22,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:22,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:23,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:24,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:25,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:26,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:27,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:28,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:28,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:29,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:30,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:31,970 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:32,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:33,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:33,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:34,491 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:06:34,491 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:06:34,492 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:06:34,493 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:06:34,664 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:06:34,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:35,665 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:06:35,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:36,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:37,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:06:37,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:06:37,666 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:06:37,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:38,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:38,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:39,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:40,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:41,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:42,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:43,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:44,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:44,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:45,387 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:06:45,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:46,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:47,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:48,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:49,675 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:06:49,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:50,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:50,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:51,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:52,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:06:52,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:06:52,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:53,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:54,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:55,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:06:55,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:56,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:57,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:58,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:06:59,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:00,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:00,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:01,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:02,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:03,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:04,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:07:04,798 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:07:04,798 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:07:04,800 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:07:05,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:05,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:05,696 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:07:06,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:07,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:07,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:07:07,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:07:07,698 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:07:08,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:09,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:10,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:11,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:11,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:12,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:13,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:14,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:15,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:15,391 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:07:16,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:17,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:17,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:17,704 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:07:18,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:19,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:20,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:21,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:22,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:22,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:22,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:07:22,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:07:23,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:24,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:25,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:26,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:27,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:27,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:28,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:29,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:29,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:07:29,907 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:07:29,907 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:07:29,908 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:07:30,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:30,721 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:07:31,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:31,722 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:07:32,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:32,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:33,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:34,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:35,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:36,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:37,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:07:37,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:07:37,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:37,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:38,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:39,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:40,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:41,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:42,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:43,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:43,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:44,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:45,395 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:07:45,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:45,732 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:07:46,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:47,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:48,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:48,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:49,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:50,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:51,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:52,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:07:52,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:07:52,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:53,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:53,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:07:54,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:55,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:56,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:57,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:58,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:59,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:07:59,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:00,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:01,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:08:01,321 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:08:01,322 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:08:01,323 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:08:01,752 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:08:01,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:02,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:03,754 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:08:03,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:04,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:04,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:05,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:06,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:07,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:08:07,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:08:07,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:08,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:09,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:09,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:10,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:11,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:12,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:13,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:14,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:15,399 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:08:15,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:15,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:15,763 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:08:16,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:17,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:18,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:19,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:20,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:20,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:21,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:22,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:08:22,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:08:22,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:23,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:24,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:25,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:25,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:26,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:27,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:28,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:29,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:30,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:31,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:08:31,210 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:08:31,210 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:08:31,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:31,211 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:08:31,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:31,787 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:08:32,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:33,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:33,789 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:08:34,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:35,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:36,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:36,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:37,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:08:37,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:08:37,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:38,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:39,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:40,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:41,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:42,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:42,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:43,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:44,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:45,404 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:08:45,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:45,796 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:08:46,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:47,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:47,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:48,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:49,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:50,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:51,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:52,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:08:52,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:08:52,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:52,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:53,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:54,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:55,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:56,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:57,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:58,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:08:58,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:08:59,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:00,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:01,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:01,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:09:01,858 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:09:01,858 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:09:01,859 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:09:02,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:02,819 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:09:03,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:03,820 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:09:04,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:04,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:05,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:06,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:07,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:09:07,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:09:07,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:08,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:09,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:09,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:10,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:11,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:12,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:13,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:14,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:15,408 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:09:15,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:15,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:16,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:17,830 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:09:17,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:18,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:19,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:20,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:20,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:21,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:22,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:09:22,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:09:22,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:23,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:24,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:25,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:26,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:26,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:27,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:28,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:29,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:30,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:31,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:31,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:32,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:33,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:09:33,207 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:09:33,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:09:33,209 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:09:33,850 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:09:33,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:34,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:35,852 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:09:35,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:36,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:37,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:09:37,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:09:37,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:37,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:38,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:39,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:40,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:41,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:42,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:42,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:43,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:44,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:45,412 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:09:45,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:46,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:47,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:47,860 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:09:47,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:48,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:49,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:50,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:51,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:52,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:09:52,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:09:52,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:52,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:53,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:54,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:55,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:56,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:57,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:58,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:09:58,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:09:59,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:00,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:01,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:02,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:03,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:10:03,689 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:10:03,689 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:10:03,690 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:10:03,879 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:10:04,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:04,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:05,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:05,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:10:06,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:07,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:07,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:10:07,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:10:08,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:09,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:09,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:10,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:11,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:12,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:13,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:14,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:14,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:15,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:15,416 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:10:16,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:17,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:17,888 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:10:18,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:19,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:19,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:20,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:21,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:22,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:22,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:10:22,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:10:23,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:24,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:25,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:25,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:26,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:27,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:28,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:29,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:30,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:30,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:31,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:32,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:33,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:33,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:10:33,250 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:10:33,250 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:10:33,252 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:10:33,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:10:34,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:35,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:35,908 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:10:36,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:36,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:37,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:37,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:10:37,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:10:38,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:39,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:40,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:41,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:42,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:42,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:43,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:44,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:45,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:45,420 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:10:46,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:47,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:47,915 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:10:48,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:48,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:49,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:50,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:51,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:52,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:52,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:10:52,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:10:53,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:53,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:54,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:55,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:56,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:57,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:58,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:10:58,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:10:59,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:00,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:01,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:02,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:03,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:03,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:03,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:11:03,920 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:11:03,920 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:11:03,922 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:11:03,933 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:11:04,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:05,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:05,934 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:11:06,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:07,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:07,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:11:07,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:11:08,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:09,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:09,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:10,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:11,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:12,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:13,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:14,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:14,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:15,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:15,425 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:11:16,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:17,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:17,941 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:11:18,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:19,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:20,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:20,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:21,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:22,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:22,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:11:22,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:11:23,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:24,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:25,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:25,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:26,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:27,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:28,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:29,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:30,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:30,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:31,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:32,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:33,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:34,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:11:34,304 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:11:34,304 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:11:34,306 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:11:34,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:34,967 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:11:35,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:35,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:35,968 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:11:36,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:37,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:37,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:11:37,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:11:38,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:39,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:40,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:40,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:41,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:42,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:43,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:44,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:45,428 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:11:45,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:45,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:46,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:47,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:47,976 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:11:48,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:49,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:50,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:51,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:51,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:52,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:52,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:11:52,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:11:53,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:54,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:55,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:56,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:56,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:11:57,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:58,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:11:59,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:00,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:01,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:01,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:02,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:12:02,256 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:12:02,257 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:12:02,258 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:12:02,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:02,999 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:12:03,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:04,000 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:12:04,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:05,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:06,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:07,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:12:07,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:12:07,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:07,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:08,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:09,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:10,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:11,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:12,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:12,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:13,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:14,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:15,432 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:12:15,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:16,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:17,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:18,009 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:12:18,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:18,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:19,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:20,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:21,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:22,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:12:22,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:12:22,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:23,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:24,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:24,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:25,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:26,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:27,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:28,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:29,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:29,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:30,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:31,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:32,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:12:32,609 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:12:32,610 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:12:32,611 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:12:33,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:33,026 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:12:34,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:34,027 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:12:35,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:35,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:36,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:37,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:37,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:12:37,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:12:38,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:39,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:40,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:40,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:41,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:42,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:43,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:44,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:45,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:45,436 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:12:45,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:46,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:46,034 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:12:47,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:48,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:49,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:50,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:50,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:51,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:52,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:52,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:12:52,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:12:53,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:54,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:55,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:55,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:12:56,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:57,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:58,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:12:59,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:00,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:01,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:01,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:01,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:13:01,809 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:13:01,810 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:13:01,811 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:13:02,058 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:13:02,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:03,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:04,060 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:13:04,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:05,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:06,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:06,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:07,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:07,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:13:07,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:13:08,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:09,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:10,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:11,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:12,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:12,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:13,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:14,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:15,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:15,439 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:13:16,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:17,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:18,068 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:13:18,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:18,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:19,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:20,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:21,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:22,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:22,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:13:22,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:13:23,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:23,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:24,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:25,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:26,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:27,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:28,300 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:28,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:29,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:30,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:31,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:32,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:33,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:13:33,161 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:13:33,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:13:33,162 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:13:33,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:34,095 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:13:34,096 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:13:34,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:34,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:35,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:36,097 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:13:36,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:37,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:37,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:13:37,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:13:38,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:39,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:39,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:40,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:41,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:42,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:43,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:44,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:45,443 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:13:45,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:45,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:46,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:47,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:48,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:49,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:50,105 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:13:50,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:50,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:51,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:52,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:52,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:13:52,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:13:53,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:54,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:55,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:56,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:56,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:13:57,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:58,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:13:59,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:00,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:01,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:01,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:02,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:03,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:04,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:04,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:14:04,639 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:14:04,640 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:14:04,640 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:14:05,123 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:14:05,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:06,124 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:14:06,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:07,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:07,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:14:07,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:14:07,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:08,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:09,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:10,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:11,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:12,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:12,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:13,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:14,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:15,447 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:14:15,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:16,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:17,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:18,131 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:14:18,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:18,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:19,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:20,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:21,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:22,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:22,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:14:22,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:14:23,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:24,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:24,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:25,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:26,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:27,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:28,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:29,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:29,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:30,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:31,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:32,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:33,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:33,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:14:33,786 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:14:33,787 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:14:33,788 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:14:34,150 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:14:34,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:35,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:35,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:36,151 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:14:36,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:37,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:37,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:14:37,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:14:38,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:39,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:40,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:40,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:41,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:42,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:43,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:44,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:45,451 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:14:45,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:45,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:46,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:47,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:48,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:49,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:50,159 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:14:50,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:50,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:51,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:52,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:52,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:14:52,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:14:53,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:54,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:55,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:55,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:14:56,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:57,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:58,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:14:59,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:00,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:00,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:01,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:02,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:03,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:04,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:15:04,559 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:15:04,559 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:15:04,560 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:15:04,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:05,176 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:15:05,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:05,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:06,177 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:15:06,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:07,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:15:07,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:15:07,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:08,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:09,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:10,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:10,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:11,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:12,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:13,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:14,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:15,455 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:15:15,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:16,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:16,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:17,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:18,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:19,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:20,186 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:15:20,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:21,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:21,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:22,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:15:22,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:15:22,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:23,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:24,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:25,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:26,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:27,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:27,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:28,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:29,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:30,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:31,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:32,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:32,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:33,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:34,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:35,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:35,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:15:35,814 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:15:35,814 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:15:35,815 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:15:36,205 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:15:36,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:37,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:15:37,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:15:37,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:38,206 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:15:38,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:38,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:39,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:40,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:41,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:42,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:43,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:43,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:44,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:45,459 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:15:45,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:46,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:47,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:48,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:49,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:49,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:50,214 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:15:50,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:51,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:52,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:15:52,634 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:15:52,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:53,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:54,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:54,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:15:55,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:56,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:57,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:58,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:59,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:15:59,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:00,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:01,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:02,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:03,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:04,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:04,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:05,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:06,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:16:06,575 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:16:06,575 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:16:06,577 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:16:06,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:07,233 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:16:07,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:16:07,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:16:07,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:08,234 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:16:08,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:09,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:10,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:10,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:11,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:12,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:13,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:14,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:15,464 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:16:15,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:15,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:16,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:17,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:18,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:19,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:21,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:21,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:22,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:22,242 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:16:22,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:16:22,633 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:16:23,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:24,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:25,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:26,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:26,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:27,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:28,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:29,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:30,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:31,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:31,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:32,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:33,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:34,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:35,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:36,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:36,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:37,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:37,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:16:37,641 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:16:38,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:39,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:39,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:16:39,725 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:16:39,725 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:16:39,726 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:16:40,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:40,258 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:16:41,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:42,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:42,259 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:16:42,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:43,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:44,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:45,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:45,468 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:16:46,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:47,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:48,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:48,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:49,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:50,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:51,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:52,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:52,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:16:52,636 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:16:53,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:53,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:16:54,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:54,267 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:16:55,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:56,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:57,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:58,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:59,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:16:59,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:00,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:01,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:02,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:03,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:04,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:05,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:05,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:06,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:07,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:07,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:17:07,642 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:17:08,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:09,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:10,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:10,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:17:10,342 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:17:10,342 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:17:10,343 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:17:10,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:11,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:11,284 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:17:12,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:12,285 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:17:13,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:14,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:15,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:15,471 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:17:16,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:16,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:17,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:18,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:19,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:20,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:21,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:22,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:22,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:17:22,639 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:17:22,779 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:23,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:24,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:24,291 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:17:25,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:26,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:27,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:27,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:28,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:29,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:30,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:31,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:32,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:33,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:33,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:34,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:35,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:36,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:37,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:37,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:17:37,643 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:17:38,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:39,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:39,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:40,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:40,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:17:40,496 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:17:40,497 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:17:40,498 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:17:41,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:41,314 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:17:42,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:42,315 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:17:43,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:44,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:44,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:45,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:45,475 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:17:46,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:47,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:48,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:49,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:50,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:50,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:51,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:52,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:52,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:17:52,643 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:17:53,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:54,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:55,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:55,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:17:56,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:56,323 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:17:57,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:58,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:17:59,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:00,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:00,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:01,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:02,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:03,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:04,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:05,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:05,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:06,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:07,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:07,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:18:07,648 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:18:08,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:08,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:18:08,949 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:18:08,949 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:18:08,950 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:18:09,343 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:18:09,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:10,344 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:18:10,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:10,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:11,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:12,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:13,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:14,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:15,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:15,479 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:18:16,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:16,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:17,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:18,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:19,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:20,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:21,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:22,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:22,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:18:22,648 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:18:22,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:23,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:24,353 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:18:24,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:25,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:26,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:27,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:27,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:28,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:29,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:30,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:31,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:32,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:33,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:33,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:34,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:35,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:36,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:37,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:37,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:18:37,660 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:18:38,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:39,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:39,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:40,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:41,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:41,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:18:41,720 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:18:41,721 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:18:41,722 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:18:42,372 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:18:42,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:43,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:44,373 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:18:44,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:45,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:45,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:45,484 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:18:46,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:47,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:48,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:49,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:50,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:50,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:51,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:52,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:52,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:18:52,654 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:18:53,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:54,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:55,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:55,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:18:56,380 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:18:56,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:57,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:58,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:18:59,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:00,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:00,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:01,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:02,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:03,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:04,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:05,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:05,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:06,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:07,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:07,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:19:07,661 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:19:08,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:09,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:10,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:10,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:11,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:11,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:19:11,758 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:19:11,758 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:19:11,760 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:19:12,403 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:19:12,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:13,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:14,404 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:19:14,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:15,487 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:19:15,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:15,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:16,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:17,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:18,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:19,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:20,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:20,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:21,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:22,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:22,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:19:22,657 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:19:23,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:24,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:25,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:25,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:26,411 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:19:26,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:27,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:28,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:29,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:30,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:30,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:31,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:32,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:33,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:34,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:35,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:35,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:36,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:37,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:37,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:19:37,659 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:19:38,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:39,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:40,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:40,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:41,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:41,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:19:41,952 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:19:41,952 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:19:41,954 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:19:42,436 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:19:42,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:43,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:44,438 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:19:44,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:45,491 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:19:45,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:46,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:46,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:47,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:48,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:49,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:50,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:51,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:51,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:52,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:52,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:19:52,659 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:19:53,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:54,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:55,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:56,444 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:19:56,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:56,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:19:57,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:58,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:19:59,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:00,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:01,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:02,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:02,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:03,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:04,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:05,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:06,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:07,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:07,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:20:07,666 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:20:08,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:08,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:09,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:20:09,503 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:20:09,504 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:20:09,505 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:20:09,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:10,466 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:20:10,467 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:20:10,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:11,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:12,468 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:20:12,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:13,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:13,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:14,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:15,495 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:20:15,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:16,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:17,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:18,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:19,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:19,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:20,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:21,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:22,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:22,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:20:22,663 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:20:23,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:24,475 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:20:24,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:25,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:25,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:26,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:27,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:28,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:29,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:30,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:30,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:31,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:32,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:33,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:34,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:35,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:35,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:36,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:37,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:37,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:20:37,671 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:20:38,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:20:38,285 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:20:38,285 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:20:38,286 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:20:38,498 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:20:38,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:39,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:40,500 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:20:40,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:41,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:41,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:42,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:43,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:44,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:45,499 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:20:45,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:46,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:47,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:47,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:48,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:49,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:50,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:51,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:52,510 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:20:52,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:20:52,670 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:20:52,709 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:53,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:53,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:54,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:55,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:56,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:57,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:58,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:20:59,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:20:59,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:00,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:01,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:02,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:03,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:04,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:04,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:05,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:06,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:07,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:21:07,686 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:21:07,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:08,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:21:08,690 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:21:08,690 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:21:08,692 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:21:09,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:09,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:09,532 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:21:09,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:10,533 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:21:10,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:11,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:12,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:13,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:14,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:15,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:15,504 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:21:15,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:16,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:17,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:18,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:19,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:20,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:21,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:21,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:22,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:21:22,671 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:21:22,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:23,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:24,541 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:21:24,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:25,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:26,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:26,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:27,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:28,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:29,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:30,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:31,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:32,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:32,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:33,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:34,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:35,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:36,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:37,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:37,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:21:37,672 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:21:37,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:38,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:39,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:39,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:21:39,982 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:21:39,982 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:21:39,984 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:21:40,559 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:21:40,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:41,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:42,560 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:21:42,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:43,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:43,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:44,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:45,507 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:21:45,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:46,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:47,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:48,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:49,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:49,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:50,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:51,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:52,566 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:21:52,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:21:52,672 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:21:52,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:53,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:54,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:21:54,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:55,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:56,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:57,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:58,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:21:59,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:00,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:00,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:01,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:02,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:03,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:04,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:05,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:05,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:06,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:07,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:22:07,674 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:22:07,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:08,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:09,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:22:09,054 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:22:09,055 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:22:09,056 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:22:09,591 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:22:09,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:10,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:10,592 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:22:10,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:11,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:12,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:13,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:14,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:15,511 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:22:15,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:15,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:16,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:17,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:18,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:19,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:20,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:21,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:21,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:22,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:22:22,674 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:22:23,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:23,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:24,599 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:22:24,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:25,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:26,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:26,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:27,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:28,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:29,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:30,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:31,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:31,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:32,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:33,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:34,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:35,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:36,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:36,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:37,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:22:37,682 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:22:37,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:38,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:39,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:40,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:22:40,330 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:22:40,330 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:22:40,332 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:22:40,625 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:22:40,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:41,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:41,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:42,628 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:22:42,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:43,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:44,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:45,515 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:22:45,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:46,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:47,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:47,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:48,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:49,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:50,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:51,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:52,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:52,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:22:52,721 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:22:52,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:53,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:54,635 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:22:54,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:55,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:56,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:57,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:22:57,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:58,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:22:59,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:00,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:01,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:02,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:03,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:03,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:04,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:05,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:06,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:07,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:23:07,723 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:23:07,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:08,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:08,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:09,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:10,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:23:10,054 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:23:10,055 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:23:10,056 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:23:10,660 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:23:10,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:11,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:12,661 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:23:12,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:13,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:14,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:14,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:15,519 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:23:15,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:16,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:17,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:18,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:19,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:20,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:20,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:21,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:22,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:23:22,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:23:22,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:23,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:24,669 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:23:24,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:25,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:25,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:26,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:27,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:28,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:29,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:30,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:31,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:31,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:32,970 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:33,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:34,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:35,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:36,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:36,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:37,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:23:37,775 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:23:37,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:38,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:23:38,054 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:23:38,054 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:23:38,056 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:23:38,687 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:23:38,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:39,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:40,688 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:23:40,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:41,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:42,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:42,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:43,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:44,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:45,522 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:23:45,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:46,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:47,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:48,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:48,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:49,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:50,694 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:23:50,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:51,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:52,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:23:52,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:23:52,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:53,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:53,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:54,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:55,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:56,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:57,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:58,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:23:59,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:23:59,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:00,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:01,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:02,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:03,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:04,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:04,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:24:04,406 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:24:04,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:24:04,408 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:24:04,720 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:24:04,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:05,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:06,722 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:24:06,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:07,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:24:07,775 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:24:07,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:08,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:09,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:09,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:10,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:11,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:12,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:13,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:14,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:15,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:15,526 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:24:15,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:16,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:17,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:18,729 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:24:18,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:19,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:20,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:20,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:21,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:22,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:24:22,775 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:24:22,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:23,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:24,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:25,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:25,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:26,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:27,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:28,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:29,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:30,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:31,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:31,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:32,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:33,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:34,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:24:34,259 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:24:34,259 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:24:34,260 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:24:34,752 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:24:34,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:35,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:36,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:36,753 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:24:36,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:37,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:24:37,775 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:24:37,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:38,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:39,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:40,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:41,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:41,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:42,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:43,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:44,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:45,530 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:24:45,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:46,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:47,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:47,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:48,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:49,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:50,762 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:24:50,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:51,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:52,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:52,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:24:52,775 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:24:52,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:53,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:54,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:55,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:56,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:57,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:58,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:24:58,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:24:59,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:00,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:01,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:02,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:03,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:04,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:05,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:06,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:07,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:07,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:25:07,786 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:25:08,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:08,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:09,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:09,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:25:09,593 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:25:09,593 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:25:09,595 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:25:09,785 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:25:10,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:10,786 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:25:11,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:12,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:13,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:14,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:14,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:15,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:15,534 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:25:16,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:17,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:18,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:19,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:20,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:20,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:21,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:22,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:22,779 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:25:22,779 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:25:22,794 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:25:23,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:24,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:25,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:25,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:26,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:27,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:28,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:29,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:30,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:31,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:31,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:32,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:33,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:34,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:35,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:36,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:36,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:37,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:37,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:25:37,246 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:25:37,247 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:25:37,248 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:25:37,779 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:25:37,780 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:25:37,815 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:25:38,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:38,816 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:25:39,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:40,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:41,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:42,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:42,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:43,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:44,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:45,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:45,538 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:25:46,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:47,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:48,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:48,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:49,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:50,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:51,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:52,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:52,779 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:25:52,779 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:25:52,825 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:25:53,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:53,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:25:54,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:55,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:56,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:57,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:58,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:59,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:25:59,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:00,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:01,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:02,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:03,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:04,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:04,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:05,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:06,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:07,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:07,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:26:07,781 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:26:08,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:08,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:26:08,383 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:26:08,384 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:26:08,386 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:26:08,844 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:26:09,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:09,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:10,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:10,846 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:26:11,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:12,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:13,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:14,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:15,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:15,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:15,542 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:26:16,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:17,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:18,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:19,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:20,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:20,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:21,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:22,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:22,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:26:22,781 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:26:22,853 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:26:23,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:24,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:25,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:25,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:26,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:27,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:28,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:29,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:30,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:31,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:31,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:32,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:33,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:34,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:35,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:36,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:36,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:37,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:37,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:26:37,789 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:26:38,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:38,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:26:38,137 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:26:38,137 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:26:38,139 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:26:38,872 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:26:38,873 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:26:39,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:40,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:40,874 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:26:41,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:41,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:42,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:43,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:44,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:45,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:45,547 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:26:46,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:47,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:47,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:48,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:49,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:50,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:51,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:52,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:52,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:52,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:26:52,785 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:26:52,882 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:26:53,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:54,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:55,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:56,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:57,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:58,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:26:58,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:26:59,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:00,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:01,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:02,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:03,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:03,105 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:04,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:05,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:06,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:07,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:07,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:27:07,792 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:27:08,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:08,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:09,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:09,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:27:09,576 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:27:09,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:27:09,578 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:27:09,916 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:27:10,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:10,917 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:27:11,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:12,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:13,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:14,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:14,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:15,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:15,550 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:27:16,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:17,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:18,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:19,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:20,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:20,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:21,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:22,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:22,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:27:22,790 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:27:22,926 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:27:23,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:24,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:25,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:25,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:26,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:27,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:28,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:29,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:30,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:31,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:31,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:32,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:33,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:34,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:35,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:36,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:36,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:37,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:37,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:27:37,795 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:27:38,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:39,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:40,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:27:40,016 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:27:40,016 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:27:40,018 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:27:40,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:40,947 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:27:40,947 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:27:41,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:41,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:42,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:42,949 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:27:43,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:44,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:45,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:45,555 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:27:46,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:46,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:47,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:48,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:49,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:50,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:51,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:51,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:52,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:52,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:27:52,795 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:27:52,957 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:27:53,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:54,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:55,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:56,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:27:56,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:57,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:58,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:27:59,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:00,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:01,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:01,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:02,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:03,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:04,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:05,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:06,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:06,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:06,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:28:06,446 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:28:06,446 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:28:06,448 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:28:06,976 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:28:07,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:07,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:28:07,795 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:28:08,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:08,978 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:28:09,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:10,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:11,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:12,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:12,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:13,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:14,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:15,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:15,559 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:28:16,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:17,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:18,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:18,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:19,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:20,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:20,987 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:28:21,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:22,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:22,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:28:22,795 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:28:23,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:23,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:24,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:25,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:26,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:27,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:28,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:29,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:29,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:30,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:31,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:32,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:33,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:34,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:34,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:34,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:28:34,621 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:28:34,621 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:28:34,623 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:28:35,013 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:28:35,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:36,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:37,014 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:28:37,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:37,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:28:37,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:28:38,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:39,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:39,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:40,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:41,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:42,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:43,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:44,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:45,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:45,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:45,563 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:28:46,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:47,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:48,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:49,024 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:28:49,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:50,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:50,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:51,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:52,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:52,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:28:52,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:28:53,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:54,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:55,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:55,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:28:56,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:57,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:58,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:28:59,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:00,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:01,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:01,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:02,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:03,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:04,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:05,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:29:05,030 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:29:05,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:29:05,031 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:29:05,044 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:29:05,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:06,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:07,046 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:29:07,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:07,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:07,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:29:07,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:29:08,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:09,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:10,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:11,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:12,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:13,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:13,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:14,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:15,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:15,567 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:29:16,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:17,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:18,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:18,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:19,053 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:29:19,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:20,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:21,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:22,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:22,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:29:22,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:29:23,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:23,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:24,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:25,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:26,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:27,491 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:28,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:29,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:29,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:30,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:31,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:32,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:32,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:29:32,800 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:29:32,800 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:29:32,803 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:29:33,077 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:29:33,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:34,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:34,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:35,078 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:29:35,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:36,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:37,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:37,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:29:37,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:29:38,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:39,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:39,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:40,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:41,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:42,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:43,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:44,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:45,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:45,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:45,571 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:29:46,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:47,086 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:29:47,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:48,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:49,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:50,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:50,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:51,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:52,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:52,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:29:52,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:29:53,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:54,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:55,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:55,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:29:56,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:57,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:58,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:29:59,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:00,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:01,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:01,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:02,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:03,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:30:03,138 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:30:03,139 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:30:03,140 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:30:03,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:04,105 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:30:04,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:05,107 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:30:05,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:06,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:06,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:07,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:07,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:30:07,796 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:30:08,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:09,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:10,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:11,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:12,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:12,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:13,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:14,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:15,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:15,575 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:30:16,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:17,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:18,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:18,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:19,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:20,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:21,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:22,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:22,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:30:22,797 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:30:23,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:23,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:24,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:25,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:26,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:28,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:29,123 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:30:29,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:34,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:36,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:37,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:39,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:44,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:44,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:45,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:30:45,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:30:45,579 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:30:46,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:50,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:30:54,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:54,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:30:54,460 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:30:55,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:30:56,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:01,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:03,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:03,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:06,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:11,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:12,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:31:12,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:31:12,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:15,583 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:31:17,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:22,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:24,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:24,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:24,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:31:24,990 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:31:25,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:26,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:27,450 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:31:27,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:27,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:28,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:29,452 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:31:29,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:30,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:31,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:32,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:32,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:33,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:34,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:35,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:36,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:37,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:38,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:38,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:39,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:39,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:31:39,990 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:31:40,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:41,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:42,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:43,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:44,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:44,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:45,589 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:31:45,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:46,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:47,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:48,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:49,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:49,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:50,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:51,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:52,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:53,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:54,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:54,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:54,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:31:54,990 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:31:55,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:56,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:57,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:58,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:31:59,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:31:59,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:00,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:01,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:02,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:03,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:04,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:05,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:05,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:06,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:07,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:08,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:09,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:09,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:32:09,990 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:32:10,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:11,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:11,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:12,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:13,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:14,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:15,591 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:32:15,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:16,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:16,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:17,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:21,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:26,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:31,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:36,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:38,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:32:38,474 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:32:38,514 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:39,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:40,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:41,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:42,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:42,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:43,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:44,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:45,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:45,595 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:32:46,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:47,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:47,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:48,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:49,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:50,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:51,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:52,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:53,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:32:53,470 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:32:53,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:53,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:32:54,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:55,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:56,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:57,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:58,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:59,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:32:59,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:00,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:01,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:02,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:03,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:04,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:04,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:05,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:06,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:07,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:08,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:33:08,470 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:33:08,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:09,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:09,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:10,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:11,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:12,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:13,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:14,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:15,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:15,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:15,599 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:33:16,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:17,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:18,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:19,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:20,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:20,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:21,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:22,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:23,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:33:23,470 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:33:23,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:24,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:25,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:25,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:26,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:27,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:28,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:29,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:30,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:31,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:31,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:32,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:33,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:34,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:35,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:36,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:36,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:37,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:38,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:33:38,470 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:33:38,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:39,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:40,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:41,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:41,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:42,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:43,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:44,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:45,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:45,604 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:33:46,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:46,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:47,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:48,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:49,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:50,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:51,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:52,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:52,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:53,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:33:53,471 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:33:53,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:54,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:55,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:56,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:57,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:57,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:33:58,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:33:59,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:00,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:01,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:02,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:03,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:03,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:04,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:05,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:06,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:07,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:08,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:34:08,471 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:34:08,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:08,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:09,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:10,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:11,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:12,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:13,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:14,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:14,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:15,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:15,606 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:34:16,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:17,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:18,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:19,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:19,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:20,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:21,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:22,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:23,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:34:23,471 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:34:23,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:24,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:34:24,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:29,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:34,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:39,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:44,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:45,615 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:34:49,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:54,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:34:59,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:04,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:04,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:04,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:35:04,851 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:35:05,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:06,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:08,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:09,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:09,739 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:35:09,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:10,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:11,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:11,742 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:35:12,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:13,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:14,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:14,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:15,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:15,614 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:35:16,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:17,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:18,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:19,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:19,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:35:19,849 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:35:19,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:20,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:21,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:21,749 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:35:22,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:23,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:24,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:25,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:25,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:26,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:27,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:28,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:29,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:30,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:30,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:31,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:32,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:33,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:34,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:34,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:35:34,859 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:35:35,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:35,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:36,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:37,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:38,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:38,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:35:38,082 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:35:38,083 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:35:38,085 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:35:38,778 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:35:39,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:39,779 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:35:40,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:41,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:41,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:42,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:43,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:44,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:45,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:45,618 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:35:46,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:47,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:47,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:48,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:49,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:49,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:35:49,859 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:35:50,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:51,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:52,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:52,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:53,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:53,788 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:35:54,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:55,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:56,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:57,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:58,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:35:58,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:35:59,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:00,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:01,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:02,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:03,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:04,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:04,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:04,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:36:04,859 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:36:05,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:06,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:07,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:08,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:09,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:09,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:10,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:11,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:12,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:12,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:36:12,348 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:36:12,348 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:36:12,350 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:36:12,805 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:36:13,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:13,806 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:36:14,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:15,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:15,623 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:36:15,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:16,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:17,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:18,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:19,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:19,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:36:19,859 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:36:20,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:20,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:21,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:22,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:23,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:24,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:25,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:25,816 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:36:26,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:26,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:27,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:28,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:29,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:30,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:31,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:32,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:32,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:33,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:34,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:34,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:36:34,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:36:35,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:36,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:37,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:37,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:38,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:39,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:40,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:41,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:42,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:43,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:36:43,016 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:36:43,016 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:36:43,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:43,018 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:36:43,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:43,841 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:36:43,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:36:44,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:45,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:45,627 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:36:45,843 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:36:46,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:47,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:48,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:48,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:49,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:49,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:36:49,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:36:50,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:51,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:52,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:53,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:53,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:36:54,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:55,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:56,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:57,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:57,852 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:36:58,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:59,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:36:59,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:00,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:01,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:02,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:03,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:04,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:04,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:37:04,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:37:04,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:05,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:06,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:07,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:08,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:09,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:10,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:10,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:11,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:12,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:13,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:14,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:14,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:37:14,866 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:37:14,867 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:37:14,868 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:37:14,876 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:37:15,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:15,631 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:37:15,878 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:37:16,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:16,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:17,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:17,879 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:37:18,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:19,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:19,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:37:19,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:37:20,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:21,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:22,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:22,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:23,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:24,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:25,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:26,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:27,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:28,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:28,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:29,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:30,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:31,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:31,890 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:37:32,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:33,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:34,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:34,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:34,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:37:34,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:37:35,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:36,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:37,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:38,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:39,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:39,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:40,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:41,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:42,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:43,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:44,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:45,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:45,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:45,635 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:37:46,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:47,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:47,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:37:47,486 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:37:47,487 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:37:47,488 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:37:47,912 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:37:48,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:49,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:49,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:37:49,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:37:49,913 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:37:50,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:51,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:51,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:52,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:53,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:54,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:55,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:56,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:57,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:37:57,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:58,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:37:59,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:00,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:01,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:02,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:02,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:03,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:03,923 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:38:04,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:04,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:38:04,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:38:05,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:06,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:07,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:08,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:08,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:09,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:10,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:11,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:12,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:13,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:14,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:14,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:15,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:15,641 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:38:16,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:17,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:18,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:19,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:38:19,247 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:38:19,247 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:38:19,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:19,249 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:38:19,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:19,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:38:19,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:38:19,943 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:38:20,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:21,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:21,945 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:38:22,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:23,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:24,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:25,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:25,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:26,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:27,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:28,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:29,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:30,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:31,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:31,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:32,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:33,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:33,954 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:38:34,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:34,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:38:34,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:38:35,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:36,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:36,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:37,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:38,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:39,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:40,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:41,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:42,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:42,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:43,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:44,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:45,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:45,642 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:38:46,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:47,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:47,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:47,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:38:47,708 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:38:47,708 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:38:47,710 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:38:47,979 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:38:48,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:49,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:49,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:38:49,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:38:49,981 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:38:50,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:51,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:52,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:53,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:53,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:54,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:55,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:56,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:57,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:58,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:38:59,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:38:59,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:00,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:01,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:01,989 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:39:02,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:03,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:04,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:04,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:04,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:39:04,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:39:05,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:06,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:07,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:08,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:09,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:10,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:10,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:11,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:12,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:13,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:14,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:15,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:15,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:15,648 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:39:16,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:17,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:18,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:18,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:39:18,764 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:39:18,765 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:39:18,767 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:39:19,008 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:39:19,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:19,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:39:19,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:39:20,009 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:39:20,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:21,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:21,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:22,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:23,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:24,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:25,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:26,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:26,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:27,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:28,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:29,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:30,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:31,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:31,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:32,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:33,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:34,021 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:39:34,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:34,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:39:34,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:39:35,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:36,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:37,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:37,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:38,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:39,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:40,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:41,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:42,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:43,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:43,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:44,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:45,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:45,651 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:39:46,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:47,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:48,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:48,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:49,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:49,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:39:49,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:39:50,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:39:50,031 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:39:50,031 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:39:50,037 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:39:50,039 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:39:50,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:51,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:52,042 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:39:52,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:53,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:54,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:54,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:39:55,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:56,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:57,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:58,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:39:59,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:00,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:00,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:01,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:02,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:03,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:04,051 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:40:04,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:04,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:40:04,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:40:05,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:06,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:06,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:07,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:08,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:09,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:10,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:11,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:11,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:12,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:13,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:14,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:15,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:15,652 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:40:16,491 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:16,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:17,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:18,492 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:19,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:19,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:40:19,864 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:40:20,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:40:20,333 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:40:20,334 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:40:20,339 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:40:20,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:21,074 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:40:21,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:21,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:22,076 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:40:22,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:23,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:24,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:25,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:26,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:27,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:27,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:28,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:29,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:30,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:31,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:32,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:32,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:33,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:34,084 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:40:34,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:34,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:40:34,862 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:40:35,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:36,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:37,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:38,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:38,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:39,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:40,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:41,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:42,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:43,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:43,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:44,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:45,661 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:40:45,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:46,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:47,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:48,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:48,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:49,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:49,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:40:49,869 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:40:50,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:40:50,586 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:40:50,587 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:40:50,592 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:40:51,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:51,109 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:40:51,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:52,109 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:40:52,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:53,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:54,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:40:54,111 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:40:54,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:55,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:56,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:57,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:59,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:40:59,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:00,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:01,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:02,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:03,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:04,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:04,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:41:04,864 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:41:05,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:05,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:06,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:06,121 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:41:07,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:08,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:09,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:10,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:10,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:11,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:12,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:13,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:14,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:15,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:15,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:15,666 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:41:16,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:17,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:18,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:19,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:19,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:41:19,868 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:41:20,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:20,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:21,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:21,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:41:21,525 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:41:21,525 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:41:21,528 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:41:22,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:22,147 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:41:23,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:24,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:24,149 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:41:25,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:26,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:26,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:27,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:28,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:29,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:30,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:31,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:32,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:32,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:33,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:34,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:34,157 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:41:34,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:41:34,868 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:41:35,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:36,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:37,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:37,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:38,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:39,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:40,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:41,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:42,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:42,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:43,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:44,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:45,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:45,665 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:41:46,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:47,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:47,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:48,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:48,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:41:48,614 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:41:48,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:41:48,616 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:41:49,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:49,177 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:41:49,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:41:49,868 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:41:50,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:50,178 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:41:51,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:52,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:53,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:53,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:41:54,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:55,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:56,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:57,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:58,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:59,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:41:59,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:00,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:01,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:02,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:02,186 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:42:03,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:04,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:04,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:04,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:42:04,868 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:42:05,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:06,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:07,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:08,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:09,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:10,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:10,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:11,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:12,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:13,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:14,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:15,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:15,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:15,667 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:42:16,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:17,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:18,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:18,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:42:18,440 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:42:18,441 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:42:18,442 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:42:19,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:19,206 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:42:19,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:42:19,869 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:42:20,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:20,207 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:42:21,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:21,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:22,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:23,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:24,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:25,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:26,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:27,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:27,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:28,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:29,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:30,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:31,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:32,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:32,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:33,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:34,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:34,217 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:42:34,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:42:34,869 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:42:35,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:36,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:37,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:38,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:38,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:39,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:40,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:41,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:42,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:43,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:44,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:44,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:45,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:45,671 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:42:46,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:47,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:48,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:49,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:49,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:42:49,665 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:42:49,665 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:42:49,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:49,667 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:42:50,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:42:50,071 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:42:50,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:50,234 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:42:51,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:52,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:52,236 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:42:53,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:54,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:55,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:55,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:42:56,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:57,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:58,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:42:59,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:00,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:01,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:01,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:02,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:03,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:04,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:05,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:43:05,074 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:43:05,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:06,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:06,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:07,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:08,247 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:43:08,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:09,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:10,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:11,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:12,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:12,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:13,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:14,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:15,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:15,675 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:43:16,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:17,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:17,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:18,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:19,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:20,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:43:20,075 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:43:20,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:21,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:22,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:23,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:23,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:23,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:43:23,924 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:43:23,925 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:43:23,926 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:43:24,273 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:43:24,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:25,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:26,274 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:43:26,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:27,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:28,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:28,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:29,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:30,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:31,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:32,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:33,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:33,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:34,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:35,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:43:35,075 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:43:35,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:36,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:37,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:38,283 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:43:38,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:38,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:39,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:40,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:41,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:42,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:43,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:44,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:44,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:45,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:45,679 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:43:46,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:47,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:48,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:49,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:49,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:50,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:43:50,079 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:43:50,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:51,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:52,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:53,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:54,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:43:54,203 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:43:54,204 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:43:54,205 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:43:54,298 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:43:54,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:55,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:43:55,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:56,300 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:43:56,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:57,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:58,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:43:59,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:00,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:00,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:01,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:02,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:03,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:04,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:05,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:44:05,079 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:44:05,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:06,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:06,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:07,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:08,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:09,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:10,310 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:44:10,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:11,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:11,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:12,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:13,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:14,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:15,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:15,683 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:44:16,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:16,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:17,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:18,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:19,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:20,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:44:20,101 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:44:20,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:21,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:22,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:22,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:23,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:24,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:25,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:26,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:44:26,023 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:44:26,024 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:44:26,025 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:44:26,328 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:44:26,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:27,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:27,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:28,330 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:44:28,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:29,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:30,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:31,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:32,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:33,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:33,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:34,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:35,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:44:35,086 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:44:35,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:36,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:37,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:38,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:39,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:39,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:40,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:41,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:42,340 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:44:42,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:43,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:44,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:44,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:45,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:45,687 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:44:46,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:47,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:48,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:49,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:49,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:50,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:44:50,086 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:44:50,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:51,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:52,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:53,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:54,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:55,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:44:55,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:56,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:57,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:44:57,369 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:44:57,370 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:44:57,371 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:44:57,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:58,366 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:44:58,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:44:59,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:00,368 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:45:00,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:01,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:01,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:02,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:03,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:04,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:05,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:45:05,086 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:45:05,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:06,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:07,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:07,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:08,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:09,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:10,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:11,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:12,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:12,375 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:45:12,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:13,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:14,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:15,690 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:45:15,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:16,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:17,692 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:17,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:18,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:19,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:20,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:45:20,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:45:20,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:21,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:22,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:23,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:23,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:24,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:25,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:26,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:27,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:28,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:28,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:45:28,439 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:45:28,440 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:45:28,441 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:45:29,396 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:45:29,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:30,397 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:45:30,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:31,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:32,399 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:45:32,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:33,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:34,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:34,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:35,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:45:35,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:45:35,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:36,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:37,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:38,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:39,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:40,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:40,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:41,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:42,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:43,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:44,408 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:45:44,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:45,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:45,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:45,694 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:45:46,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:47,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:48,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:49,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:50,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:45:50,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:45:50,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:51,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:51,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:52,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:53,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:54,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:55,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:56,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:45:56,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:57,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:58,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:45:59,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:45:59,517 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:45:59,518 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:45:59,519 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:46:00,427 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:46:00,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:01,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:01,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:02,428 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:46:02,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:03,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:04,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:05,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:46:05,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:46:05,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:06,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:07,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:07,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:08,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:09,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:10,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:11,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:12,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:13,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:13,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:14,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:15,698 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:46:15,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:16,435 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:46:16,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:17,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:18,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:18,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:19,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:20,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:46:20,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:46:20,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:21,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:22,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:23,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:24,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:24,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:25,709 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:26,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:27,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:28,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:29,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:29,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:30,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:31,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:46:31,531 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:46:31,531 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:46:31,533 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:46:31,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:32,454 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:46:32,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:33,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:34,456 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:46:34,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:34,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:35,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:46:35,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:46:35,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:36,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:37,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:38,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:39,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:40,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:40,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:41,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:42,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:43,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:44,468 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:46:44,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:45,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:45,702 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:46:45,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:46,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:47,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:48,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:49,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:50,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:46:50,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:46:50,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:51,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:51,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:52,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:53,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:54,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:55,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:56,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:46:56,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:57,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:58,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:46:59,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:46:59,614 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:46:59,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:46:59,615 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:47:00,485 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:47:00,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:01,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:01,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:02,487 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:47:02,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:03,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:04,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:05,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:47:05,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:47:05,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:06,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:07,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:07,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:08,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:09,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:10,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:11,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:12,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:12,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:13,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:14,495 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:47:14,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:15,706 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:47:15,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:16,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:17,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:17,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:18,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:19,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:20,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:47:20,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:47:20,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:21,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:22,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:23,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:23,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:24,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:25,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:26,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:27,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:28,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:28,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:29,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:30,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:30,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:47:30,791 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:47:30,792 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:47:30,793 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:47:31,514 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:47:31,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:32,515 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:47:32,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:33,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:34,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:34,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:35,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:47:35,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:47:35,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:36,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:37,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:38,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:39,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:39,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:40,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:41,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:42,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:43,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:44,523 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:47:44,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:44,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:45,711 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:47:45,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:46,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:47,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:48,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:49,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:49,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:50,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:47:50,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:47:50,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:51,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:52,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:53,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:54,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:55,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:47:55,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:56,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:57,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:58,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:47:59,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:00,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:48:00,080 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:48:00,080 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:48:00,081 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:48:00,543 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:48:00,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:01,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:01,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:02,545 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:48:02,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:03,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:04,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:05,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:48:05,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:48:05,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:06,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:06,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:07,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:08,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:09,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:10,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:11,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:11,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:12,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:13,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:14,554 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:48:14,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:15,714 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:48:15,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:16,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:16,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:17,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:18,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:19,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:20,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:48:20,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:48:20,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:21,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:22,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:22,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:23,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:24,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:25,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:26,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:27,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:27,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:28,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:29,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:30,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:48:30,517 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:48:30,518 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:48:30,519 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:48:30,581 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:48:30,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:31,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:32,582 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:48:32,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:32,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:33,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:34,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:35,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:48:35,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:48:35,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:36,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:37,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:38,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:38,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:39,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:40,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:41,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:42,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:43,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:44,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:44,589 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:48:44,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:45,718 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:48:45,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:46,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:47,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:48,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:49,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:49,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:50,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:48:50,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:48:50,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:51,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:52,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:53,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:54,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:55,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:48:55,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:56,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:57,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:58,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:48:59,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:00,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:49:00,121 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:49:00,121 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:49:00,123 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:49:00,607 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:49:00,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:01,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:01,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:02,609 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:49:02,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:03,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:04,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:05,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:49:05,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:49:05,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:06,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:07,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:07,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:08,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:09,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:10,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:11,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:12,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:12,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:13,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:14,617 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:49:14,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:15,722 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:49:15,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:16,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:17,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:18,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:18,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:19,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:20,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:49:20,088 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:49:20,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:21,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:22,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:23,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:24,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:24,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:25,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:26,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:27,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:28,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:29,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:49:29,498 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:49:29,499 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:49:29,500 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:49:29,641 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:49:30,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:30,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:30,641 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:49:31,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:32,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:32,643 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:49:33,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:34,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:35,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:49:35,089 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:49:35,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:35,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:36,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:37,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:38,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:39,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:40,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:40,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:41,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:42,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:43,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:44,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:45,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:45,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:45,726 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:49:46,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:46,652 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:49:47,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:48,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:49,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:50,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:49:50,089 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:49:50,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:51,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:51,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:52,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:53,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:54,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:55,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:56,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:56,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:49:57,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:58,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:49:59,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:00,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:01,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:02,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:50:02,445 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:50:02,445 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:50:02,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:02,447 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:50:02,671 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:50:02,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:03,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:04,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:04,673 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:50:05,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:50:05,089 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:50:05,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:06,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:07,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:07,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:08,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:09,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:10,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:11,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:12,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:12,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:13,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:14,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:15,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:15,731 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:50:16,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:17,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:18,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:18,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:18,683 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:50:19,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:20,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:50:20,089 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:50:20,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:21,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:22,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:23,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:23,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:24,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:25,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:26,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:27,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:28,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:29,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:29,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:30,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:31,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:32,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:33,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:34,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:34,692 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:35,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:50:35,096 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:50:35,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:35,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:50:35,963 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:50:35,963 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:50:35,964 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:50:36,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:36,705 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:50:37,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:38,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:38,707 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:50:39,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:40,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:40,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:41,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:42,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:43,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:44,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:45,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:45,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:45,734 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:50:46,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:47,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:48,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:49,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:50,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:50:50,096 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:50:50,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:50,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:50,716 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:50:51,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:52,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:53,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:54,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:55,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:56,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:56,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:50:57,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:58,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:50:59,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:00,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:01,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:01,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:02,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:03,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:04,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:05,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:51:05,096 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:51:05,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:06,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:06,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:06,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:51:06,811 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:51:06,812 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:51:06,813 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:51:07,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:07,732 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:51:08,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:08,733 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:51:09,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:10,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:11,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:12,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:12,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:13,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:14,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:15,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:15,738 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:51:16,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:17,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:17,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:18,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:19,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:20,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:51:20,097 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:51:20,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:20,748 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:51:21,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:22,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:23,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:23,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:24,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:25,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:26,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:27,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:28,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:29,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:29,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:30,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:31,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:32,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:33,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:34,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:34,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:35,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:51:35,097 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:51:35,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:36,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:51:36,350 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:51:36,350 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:51:36,351 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:51:36,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:36,767 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:51:37,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:38,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:38,768 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:51:39,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:39,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:40,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:41,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:42,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:43,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:44,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:45,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:45,742 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:51:45,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:46,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:47,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:48,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:49,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:50,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:51:50,097 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:51:50,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:50,776 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:51:51,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:51,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:52,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:53,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:54,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:55,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:56,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:51:56,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:57,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:58,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:51:59,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:00,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:01,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:01,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:02,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:03,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:04,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:05,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:52:05,099 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:52:05,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:52:05,534 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:52:05,534 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:52:05,535 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:52:05,804 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:52:05,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:06,805 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:52:06,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:06,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:07,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:08,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:09,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:10,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:11,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:12,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:12,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:13,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:14,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:15,746 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:52:15,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:16,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:17,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:18,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:18,813 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:52:18,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:19,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:20,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:52:20,099 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:52:20,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:21,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:22,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:23,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:23,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:24,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:25,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:26,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:27,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:28,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:28,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:29,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:30,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:31,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:32,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:33,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:33,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:34,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:52:34,360 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:52:34,360 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:52:34,362 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:52:34,840 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:52:34,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:35,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:52:35,099 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:52:35,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:36,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:52:36,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:37,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:38,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:39,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:39,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:40,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:41,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:42,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:43,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:44,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:45,751 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:52:45,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:45,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:46,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:47,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:48,851 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:52:48,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:49,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:50,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:52:50,099 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:52:50,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:51,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:52,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:53,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:54,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:55,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:55,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:52:56,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:57,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:58,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:52:59,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:00,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:00,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:01,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:02,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:03,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:03,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:53:03,112 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:53:03,112 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:53:03,114 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:53:03,872 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:53:04,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:04,873 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:53:05,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:05,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:53:05,100 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:53:06,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:06,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:07,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:08,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:09,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:10,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:11,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:12,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:12,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:13,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:14,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:15,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:15,754 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:53:16,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:17,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:18,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:18,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:18,883 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:53:19,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:20,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:53:20,100 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:53:20,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:21,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:22,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:23,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:23,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:24,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:25,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:26,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:27,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:28,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:29,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:29,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:30,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:31,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:32,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:33,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:34,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:34,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:35,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:53:35,103 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:53:35,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:36,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:53:36,112 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:53:36,112 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:53:36,114 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:53:36,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:36,902 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:53:37,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:38,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:38,903 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:53:39,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:40,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:40,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:41,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:42,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:43,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:44,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:45,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:45,758 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:53:45,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:46,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:47,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:48,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:49,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:50,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:53:50,103 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:53:50,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:50,911 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:53:50,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:51,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:52,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:53,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:54,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:55,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:55,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:53:56,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:57,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:58,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:53:59,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:00,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:00,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:01,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:02,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:03,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:04,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:05,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:54:05,107 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:54:05,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:06,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:54:06,244 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:54:06,244 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:54:06,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:06,246 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:54:06,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:06,931 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:54:07,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:08,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:08,933 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:54:09,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:10,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:11,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:11,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:12,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:13,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:14,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:15,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:15,762 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:54:16,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:17,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:17,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:18,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:19,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:20,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:54:20,107 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:54:20,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:21,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:22,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:22,943 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:54:22,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:23,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:24,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:25,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:26,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:27,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:28,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:28,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:29,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:30,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:31,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:32,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:33,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:33,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:34,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:35,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:54:35,107 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:54:35,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:36,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:37,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:38,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:38,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:39,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:40,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:54:40,060 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:54:40,060 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:54:40,062 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:54:40,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:40,964 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:54:40,965 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:54:41,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:42,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:42,966 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:54:43,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:43,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:44,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:45,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:45,765 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:54:46,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:47,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:48,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:49,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:49,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:50,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:54:50,108 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:54:50,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:51,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:52,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:52,973 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:54:53,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:54,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:55,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:54:55,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:56,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:57,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:58,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:54:59,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:00,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:00,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:01,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:02,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:03,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:04,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:05,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:55:05,108 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:55:05,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:05,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:06,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:07,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:55:07,178 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:55:07,179 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:55:07,180 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:55:07,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:07,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:55:08,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:08,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:55:09,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:10,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:11,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:11,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:12,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:13,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:14,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:15,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:15,770 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:55:16,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:17,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:17,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:18,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:19,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:20,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:55:20,108 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:55:20,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:21,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:22,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:23,006 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:55:23,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:23,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:24,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:25,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:26,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:27,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:28,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:28,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:29,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:30,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:31,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:32,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:33,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:33,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:34,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:35,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:55:35,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:55:35,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:36,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:37,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:38,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:38,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:39,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:55:39,023 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:55:39,023 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:55:39,025 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:55:39,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:40,023 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:55:40,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:41,024 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:55:41,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:42,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:43,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:44,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:44,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:45,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:45,773 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:55:46,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:47,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:48,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:49,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:50,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:55:50,111 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:55:50,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:50,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:51,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:52,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:53,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:54,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:55,034 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:55:55,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:55:55,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:56,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:57,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:58,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:55:59,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:00,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:00,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:01,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:02,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:03,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:04,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:05,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:56:05,118 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:56:05,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:05,978 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:06,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:07,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:08,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:09,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:10,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:56:10,444 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:56:10,445 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:56:10,446 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:56:10,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:11,053 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:56:11,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:11,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:12,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:13,054 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:56:13,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:14,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:15,777 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:56:15,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:16,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:17,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:17,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:18,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:19,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:20,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:56:20,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:56:20,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:21,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:22,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:22,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:23,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:24,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:25,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:26,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:27,064 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:56:27,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:27,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:28,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:29,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:30,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:31,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:32,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:32,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:33,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:34,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:35,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:56:35,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:56:35,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:36,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:37,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:37,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:38,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:39,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:40,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:41,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:42,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:42,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:43,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:56:43,375 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:56:43,375 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:56:43,377 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:56:43,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:44,092 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:56:44,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:45,093 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:56:45,781 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:56:45,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:46,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:47,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:48,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:48,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:49,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:50,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:56:50,117 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:56:50,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:51,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:52,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:53,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:54,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:54,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:56:55,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:56,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:57,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:58,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:56:59,104 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:56:59,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:00,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:00,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:01,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:02,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:03,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:04,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:05,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:57:05,117 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:57:05,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:05,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:06,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:07,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:08,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:09,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:10,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:10,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:11,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:12,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:12,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:57:12,958 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:57:12,958 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:57:12,960 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:57:13,129 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:57:13,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:14,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:15,131 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:57:15,785 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:57:15,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:15,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:16,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:17,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:18,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:19,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:20,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:57:20,117 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:57:20,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:21,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:21,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:22,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:23,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:24,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:25,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:26,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:26,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:27,140 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:57:27,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:28,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:29,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:30,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:31,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:32,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:32,979 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:33,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:34,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:35,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:57:35,134 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:57:35,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:36,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:37,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:37,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:38,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:39,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:40,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:57:40,741 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:57:40,742 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:57:40,743 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:57:41,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:41,161 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:57:42,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:43,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:43,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:43,162 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:57:44,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:45,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:45,789 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:57:46,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:47,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:48,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:49,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:49,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:50,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:57:50,122 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:57:50,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:51,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:52,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:53,171 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:57:53,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:54,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:54,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:57:55,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:56,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:57,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:58,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:57:59,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:00,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:00,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:01,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:02,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:03,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:04,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:05,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:58:05,134 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:58:05,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:05,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:06,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:07,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:08,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:58:08,066 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:58:08,066 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:58:08,067 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:58:08,195 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:58:08,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:09,196 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:58:09,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:10,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:10,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:11,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:12,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:13,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:14,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:15,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:15,793 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:58:15,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:16,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:17,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:18,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:19,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:20,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:58:20,131 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:58:20,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:21,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:21,204 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:58:21,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:22,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:23,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:24,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:25,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:26,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:26,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:27,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:28,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:29,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:30,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:31,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:32,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:32,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:33,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:34,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:35,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:58:35,136 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:58:35,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:36,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:37,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:37,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:37,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:58:37,845 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:58:37,845 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:58:37,846 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:58:38,226 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:58:38,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:39,228 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:58:39,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:40,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:41,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:42,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:43,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:43,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:44,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:45,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:45,796 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:58:46,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:47,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:48,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:49,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:49,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:50,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:58:50,132 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:58:50,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:51,236 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:58:51,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:52,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:53,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:54,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:54,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:55,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:56,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:57,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:58,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:58:59,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:58:59,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:00,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:01,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:02,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:03,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:04,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:04,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:05,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:59:05,140 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:59:05,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:06,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:07,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:59:07,371 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:59:07,372 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:59:07,373 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:59:07,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:08,257 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:59:08,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:09,258 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:59:09,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:09,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:10,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:11,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:12,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:13,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:14,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:15,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:15,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:15,799 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:59:16,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:17,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:18,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:19,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:20,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:59:20,137 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:59:20,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:20,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:21,266 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:59:21,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:22,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:23,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:24,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:25,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:26,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:26,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:27,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:28,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:29,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:30,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:31,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:31,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:32,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:33,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:34,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:35,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:59:35,146 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:59:35,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:36,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:37,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:37,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 09:59:37,299 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 09:59:37,299 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 09:59:37,301 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 09:59:37,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:38,284 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 09:59:38,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:39,285 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:59:39,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:40,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:41,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:42,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:42,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:43,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:44,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:45,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:45,803 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 09:59:46,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:47,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:48,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:48,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:49,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:50,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 09:59:50,141 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 09:59:50,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:51,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:52,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:53,298 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 09:59:53,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:53,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:54,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:55,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:56,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:57,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:58,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 09:59:59,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 09:59:59,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:00,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:01,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:02,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:03,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:04,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:04,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:05,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:00:05,145 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:00:05,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:06,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:07,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:08,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:09,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:09,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:09,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:00:09,856 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:00:09,856 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:00:09,858 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:00:10,325 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:00:10,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:11,326 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:00:11,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:12,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:13,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:14,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:15,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:15,807 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:00:15,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:16,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:17,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:18,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:19,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:20,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:00:20,143 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:00:20,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:20,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:21,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:22,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:23,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:24,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:25,335 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:00:25,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:26,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:26,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:27,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:28,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:29,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:30,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:31,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:31,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:32,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:33,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:34,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:35,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:00:35,143 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:00:35,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:36,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:36,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:37,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:38,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:39,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:40,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:41,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:00:41,596 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:00:41,597 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:00:41,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:41,598 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:00:42,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:42,355 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:00:43,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:43,356 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:00:44,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:45,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:45,811 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:00:46,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:47,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:47,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:48,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:49,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:50,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:50,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:00:50,143 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:00:51,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:52,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:53,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:53,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:54,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:55,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:55,368 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:00:56,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:57,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:58,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:00:58,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:00:59,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:00,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:01,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:02,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:03,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:03,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:04,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:05,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:05,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:01:05,144 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:01:06,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:07,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:08,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:08,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:09,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:10,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:11,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:11,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:01:11,205 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:01:11,205 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:01:11,207 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:01:11,396 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:01:12,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:13,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:13,398 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:01:13,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:14,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:15,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:15,815 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:01:16,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:17,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:18,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:19,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:19,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:20,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:20,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:01:20,144 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:01:21,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:22,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:23,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:24,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:24,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:25,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:25,407 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:01:26,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:27,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:28,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:29,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:30,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:30,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:31,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:32,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:33,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:34,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:35,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:35,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:01:35,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:01:35,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:36,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:37,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:38,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:39,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:39,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:01:39,769 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:01:39,770 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:01:39,771 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:01:40,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:40,426 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:01:41,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:41,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:41,427 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:01:42,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:43,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:44,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:45,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:45,819 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:01:46,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:47,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:47,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:48,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:49,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:50,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:01:50,149 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:01:50,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:51,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:52,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:53,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:53,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:54,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:55,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:56,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:57,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:57,439 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:01:58,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:01:58,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:01:59,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:00,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:01,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:02,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:03,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:04,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:04,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:05,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:02:05,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:02:05,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:06,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:07,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:08,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:09,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:09,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:10,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:11,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:12,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:13,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:14,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:14,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:14,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:02:14,645 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:02:14,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:02:14,647 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:02:15,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:15,469 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:02:15,470 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:02:15,823 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:02:16,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:17,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:17,470 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:02:18,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:19,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:20,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:02:20,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:02:20,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:20,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:21,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:22,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:23,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:24,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:25,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:26,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:26,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:27,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:27,478 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:02:28,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:29,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:30,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:31,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:31,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:32,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:33,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:34,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:35,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:02:35,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:02:35,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:36,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:37,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:37,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:38,300 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:39,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:40,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:41,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:42,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:42,301 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:43,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:43,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:02:43,813 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:02:43,814 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:02:43,815 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:02:44,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:44,498 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:02:45,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:45,499 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:02:45,827 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:02:46,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:47,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:48,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:48,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:49,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:50,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:02:50,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:02:50,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:51,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:52,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:53,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:54,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:54,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:55,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:56,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:57,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:57,508 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:02:58,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:02:59,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:02:59,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:00,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:01,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:02,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:03,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:04,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:05,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:03:05,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:03:05,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:05,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:06,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:07,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:08,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:09,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:10,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:11,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:11,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:12,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:13,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:03:13,128 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:03:13,128 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:03:13,129 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:03:13,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:13,533 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:03:14,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:15,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:15,535 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:03:15,831 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:03:16,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:16,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:17,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:18,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:19,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:20,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:03:20,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:03:20,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:21,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:21,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:22,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:23,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:24,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:25,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:26,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:27,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:27,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:27,545 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:03:28,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:29,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:30,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:31,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:32,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:32,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:33,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:34,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:35,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:03:35,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:03:35,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:36,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:37,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:37,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:38,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:39,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:40,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:41,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:42,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:42,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:42,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:03:42,915 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:03:42,916 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:03:42,917 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:03:43,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:43,570 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:03:43,570 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:03:44,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:45,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:45,571 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:03:45,835 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:03:46,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:47,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:47,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:48,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:49,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:50,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:03:50,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:03:50,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:51,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:52,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:53,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:53,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:54,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:55,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:56,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:57,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:58,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:58,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:03:59,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:03:59,581 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:04:00,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:01,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:02,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:03,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:04,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:04,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:05,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:04:05,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:04:05,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:06,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:07,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:08,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:09,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:10,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:10,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:11,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:12,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:13,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:14,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:15,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:15,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:04:15,544 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:04:15,544 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:04:15,546 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:04:15,604 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:04:15,840 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:04:15,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:16,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:17,607 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:04:17,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:18,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:19,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:20,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:04:20,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:04:20,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:20,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:21,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:22,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:23,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:24,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:25,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:26,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:26,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:27,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:28,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:29,615 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:04:29,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:30,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:31,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:31,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:32,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:33,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:34,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:35,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:04:35,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:04:35,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:36,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:37,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:37,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:38,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:39,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:40,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:41,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:42,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:42,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:43,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:44,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:45,846 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:04:45,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:46,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:04:46,366 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:04:46,366 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:04:46,368 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:04:46,647 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:04:46,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:47,648 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:04:47,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:47,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:48,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:49,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:50,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:04:50,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:04:50,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:51,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:52,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:53,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:53,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:54,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:55,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:56,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:58,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:58,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:04:59,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:04:59,655 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:04:59,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:00,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:01,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:02,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:03,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:04,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:04,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:05,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:05:05,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:05:05,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:06,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:07,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:08,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:09,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:10,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:10,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:11,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:12,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:13,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:14,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:15,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:15,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:05:15,243 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:05:15,243 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:05:15,245 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:05:15,678 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:05:15,849 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:05:15,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:16,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:17,680 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:05:17,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:18,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:19,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:20,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:05:20,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:05:20,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:20,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:21,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:22,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:23,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:24,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:25,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:26,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:26,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:28,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:29,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:29,688 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:05:30,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:31,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:31,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:32,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:33,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:34,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:35,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:35,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:05:35,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:05:36,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:36,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:37,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:38,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:39,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:40,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:41,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:41,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:42,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:43,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:44,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:45,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:45,854 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:05:45,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:05:45,918 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:05:45,918 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:05:45,920 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:05:46,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:46,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:46,710 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:05:47,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:47,711 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:05:48,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:49,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:50,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:50,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:05:50,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:05:51,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:52,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:52,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:53,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:54,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:55,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:56,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:57,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:57,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:05:58,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:59,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:05:59,721 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:06:00,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:01,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:02,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:02,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:03,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:04,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:05,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:05,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:06:05,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:06:06,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:07,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:08,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:08,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:09,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:10,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:11,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:12,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:13,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:13,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:14,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:15,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:15,856 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:06:16,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:16,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:06:16,760 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:06:16,761 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:06:16,762 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:06:17,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:17,744 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:06:17,744 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:06:18,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:19,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:19,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:20,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:06:20,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:06:20,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:21,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:22,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:23,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:24,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:24,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:25,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:26,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:27,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:28,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:29,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:29,753 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:06:30,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:30,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:31,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:32,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:33,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:34,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:35,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:06:35,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:06:35,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:36,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:36,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:37,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:38,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:39,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:40,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:41,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:41,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:42,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:43,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:44,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:45,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:45,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:06:45,645 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:06:45,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:06:45,646 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:06:45,776 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:06:45,860 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:06:46,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:47,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:47,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:47,778 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:06:48,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:49,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:50,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:06:50,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:06:50,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:51,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:52,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:53,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:53,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:54,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:55,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:56,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:57,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:58,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:58,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:06:59,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:06:59,787 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:07:00,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:01,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:02,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:03,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:04,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:04,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:05,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:07:05,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:07:05,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:06,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:07,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:08,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:09,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:09,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:10,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:11,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:12,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:13,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:14,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:14,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:07:14,883 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:07:14,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:07:14,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:14,885 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:07:15,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:15,806 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:07:15,806 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:07:15,864 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:07:16,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:17,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:18,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:19,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:20,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:07:20,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:07:20,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:20,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:21,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:22,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:23,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:24,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:25,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:25,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:26,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:27,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:27,815 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:07:28,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:29,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:30,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:30,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:31,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:32,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:33,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:34,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:35,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:07:35,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:07:35,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:36,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:36,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:37,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:38,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:39,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:40,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:41,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:41,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:42,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:43,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:43,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:07:43,769 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:07:43,770 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:07:43,772 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:07:43,832 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:07:44,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:45,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:45,835 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:07:45,868 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:07:46,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:46,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:47,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:48,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:49,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:50,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:07:50,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:07:50,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:51,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:52,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:52,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:53,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:54,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:55,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:56,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:57,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:57,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:07:57,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:07:58,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:07:59,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:00,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:01,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:02,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:03,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:03,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:04,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:05,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:08:05,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:08:05,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:06,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:07,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:08,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:08,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:09,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:10,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:11,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:12,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:13,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:13,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:14,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:08:14,082 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:08:14,083 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:08:14,084 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:08:14,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:14,867 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:08:15,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:15,867 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:08:15,873 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:08:16,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:17,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:18,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:19,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:19,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:20,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:08:20,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:08:20,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:21,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:22,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:23,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:24,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:25,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:25,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:26,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:27,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:27,875 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:08:28,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:29,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:30,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:30,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:31,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:32,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:33,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:34,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:35,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:08:35,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:08:35,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:36,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:36,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:37,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:38,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:39,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:40,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:41,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:41,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:42,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:43,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:44,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:45,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:08:45,084 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:08:45,084 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:08:45,086 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:08:45,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:45,877 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:08:45,894 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:08:45,895 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:08:46,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:47,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:47,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:47,896 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:08:48,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:49,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:50,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:08:50,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:08:50,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:51,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:52,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:53,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:53,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:54,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:55,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:56,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:57,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:58,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:08:58,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:08:59,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:00,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:01,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:01,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:09:02,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:03,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:03,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:04,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:05,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:09:05,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:09:05,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:06,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:07,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:08,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:08,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:09,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:10,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:11,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:12,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:13,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:13,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:14,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:15,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:15,884 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:09:16,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:17,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:09:17,608 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:09:17,608 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:09:17,610 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:09:17,922 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:09:18,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:18,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:19,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:19,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:19,924 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:09:20,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:09:20,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:09:20,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:21,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:22,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:23,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:24,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:24,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:25,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:26,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:27,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:28,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:29,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:30,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:30,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:31,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:31,932 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:09:32,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:33,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:34,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:35,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:09:35,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:09:35,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:36,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:36,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:37,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:38,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:39,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:40,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:41,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:41,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:42,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:43,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:44,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:45,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:45,887 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:09:46,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:46,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:47,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:48,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:09:48,678 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:09:48,679 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:09:48,681 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:09:48,950 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:09:49,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:49,951 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:09:50,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:50,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:09:50,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:09:51,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:52,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:52,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:53,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:54,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:55,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:56,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:57,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:58,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:09:58,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:09:59,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:00,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:01,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:01,959 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:10:02,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:03,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:03,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:04,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:05,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:05,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:10:05,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:10:06,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:07,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:08,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:09,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:09,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:10,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:11,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:12,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:13,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:14,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:14,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:15,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:15,895 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:10:16,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:17,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:18,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:19,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:19,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:10:19,642 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:10:19,643 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:10:19,644 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:10:19,978 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:10:20,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:20,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:20,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:10:20,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:10:21,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:21,980 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:10:22,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:23,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:24,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:25,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:25,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:26,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:27,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:28,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:29,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:30,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:31,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:31,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:32,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:33,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:33,988 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:10:34,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:35,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:10:35,154 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:10:35,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:36,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:36,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:37,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:38,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:39,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:40,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:41,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:41,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:42,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:43,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:44,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:45,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:45,892 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:10:46,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:46,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:47,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:48,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:49,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:50,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:10:50,160 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:10:50,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:51,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:52,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:52,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:10:52,628 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:10:52,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:10:52,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:52,629 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:10:53,013 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:10:53,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:54,014 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:10:54,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:55,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:56,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:57,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:57,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:10:58,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:10:59,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:00,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:01,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:02,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:02,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:03,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:04,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:05,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:11:05,156 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:11:05,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:06,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:07,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:08,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:08,024 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:11:08,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:09,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:10,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:11,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:12,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:13,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:13,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:14,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:15,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:15,895 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:11:16,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:17,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:18,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:18,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:19,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:20,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:11:20,159 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:11:20,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:21,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:22,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:23,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:23,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:24,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:24,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:11:24,747 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:11:24,748 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:11:24,749 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:11:25,043 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:11:25,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:26,044 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:11:26,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:27,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:28,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:29,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:29,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:30,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:31,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:32,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:33,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:34,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:34,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:35,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:11:35,157 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:11:35,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:36,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:37,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:38,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:39,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:40,054 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:11:40,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:40,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:41,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:42,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:43,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:44,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:45,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:45,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:45,899 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:11:46,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:47,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:48,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:49,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:50,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:11:50,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:11:50,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:51,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:51,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:52,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:53,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:54,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:54,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:11:54,522 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:11:54,523 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:11:54,524 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:11:55,074 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:11:55,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:56,075 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:11:56,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:56,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:11:57,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:58,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:11:59,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:00,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:01,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:02,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:02,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:03,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:04,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:05,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:12:05,160 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:12:05,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:06,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:07,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:07,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:08,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:09,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:10,084 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:12:10,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:11,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:12,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:12,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:13,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:14,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:15,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:15,903 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:12:16,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:17,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:18,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:18,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:19,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:20,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:12:20,160 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:12:20,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:21,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:22,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:23,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:23,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:24,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:25,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:26,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:27,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:27,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:12:27,825 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:12:27,826 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:12:27,827 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:12:28,111 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:12:28,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:29,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:29,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:30,112 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:12:30,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:31,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:32,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:33,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:34,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:34,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:35,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:12:35,160 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:12:35,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:36,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:37,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:38,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:39,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:40,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:40,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:41,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:42,121 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:12:42,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:43,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:44,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:45,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:45,907 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:12:45,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:46,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:47,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:48,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:49,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:50,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:12:50,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:12:50,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:51,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:51,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:52,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:53,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:54,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:55,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:56,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:56,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:12:57,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:58,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:12:58,282 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:12:58,283 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:12:58,284 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:12:59,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:12:59,142 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:13:00,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:00,143 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:13:01,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:01,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:02,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:03,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:04,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:05,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:05,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:13:05,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:13:06,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:07,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:07,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:08,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:09,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:10,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:11,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:12,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:12,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:13,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:14,153 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:13:14,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:15,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:15,910 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:13:16,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:17,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:17,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:18,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:19,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:20,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:13:20,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:13:20,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:21,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:22,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:23,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:23,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:24,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:25,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:26,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:27,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:28,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:28,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:29,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:30,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:30,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:13:30,706 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:13:30,706 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:13:30,707 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:13:31,175 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:13:31,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:32,176 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:13:32,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:33,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:34,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:34,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:35,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:13:35,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:13:35,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:36,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:37,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:38,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:39,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:40,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:40,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:41,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:42,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:43,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:44,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:45,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:45,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:45,914 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:13:46,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:46,189 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:13:47,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:48,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:49,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:50,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:13:50,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:13:50,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:51,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:51,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:52,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:53,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:54,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:55,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:56,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:56,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:13:57,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:58,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:13:59,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:00,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:01,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:01,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:02,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:02,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:14:02,208 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:14:02,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:14:02,209 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:14:02,212 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:14:03,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:04,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:05,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:14:05,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:14:05,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:05,215 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:14:06,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:06,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:07,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:08,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:09,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:10,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:11,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:12,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:12,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:13,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:14,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:15,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:15,918 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:14:16,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:17,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:17,223 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:14:18,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:18,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:19,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:20,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:14:20,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:14:20,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:21,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:22,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:23,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:23,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:24,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:25,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:26,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:27,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:28,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:28,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:29,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:30,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:31,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:32,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:33,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:33,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:14:33,387 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:14:33,387 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:14:33,389 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:14:34,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:34,238 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:14:34,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:35,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:14:35,161 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:14:35,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:35,238 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:14:36,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:37,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:38,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:39,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:39,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:40,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:41,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:42,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:43,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:44,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:44,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:45,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:45,922 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:14:46,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:47,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:48,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:49,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:49,248 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:14:50,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:14:50,162 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:14:50,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:50,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:51,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:52,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:53,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:54,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:55,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:55,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:14:56,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:57,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:58,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:14:59,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:00,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:00,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:01,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:02,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:03,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:04,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:05,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:15:05,167 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:15:05,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:06,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:06,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:06,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:15:06,426 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:15:06,426 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:15:06,428 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:15:07,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:07,267 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:15:08,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:09,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:09,269 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:15:10,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:11,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:11,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:12,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:13,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:14,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:15,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:15,926 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:15:16,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:17,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:17,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:18,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:19,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:20,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:15:20,162 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:15:20,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:21,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:22,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:22,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:23,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:23,280 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:15:24,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:25,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:26,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:27,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:28,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:28,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:29,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:30,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:31,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:32,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:33,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:33,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:34,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:35,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:15:35,162 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:15:35,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:36,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:37,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:38,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:38,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:15:38,618 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:15:38,618 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:15:38,619 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:15:39,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:39,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:39,302 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:15:40,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:41,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:41,305 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:15:42,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:43,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:44,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:44,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:45,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:45,931 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:15:46,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:47,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:48,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:49,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:50,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:15:50,162 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:15:50,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:50,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:51,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:52,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:53,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:54,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:55,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:55,315 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:15:56,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:15:56,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:57,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:58,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:15:59,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:00,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:01,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:01,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:02,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:03,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:04,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:05,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:16:05,174 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:16:05,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:06,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:06,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:07,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:08,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:09,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:10,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:10,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:16:10,773 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:16:10,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:16:10,775 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:16:11,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:11,338 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:16:12,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:12,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:13,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:13,339 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:16:14,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:15,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:15,933 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:16:16,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:17,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:18,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:18,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:19,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:20,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:16:20,164 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:16:20,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:21,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:22,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:23,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:24,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:24,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:25,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:26,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:27,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:27,349 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:16:28,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:29,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:29,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:30,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:31,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:32,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:33,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:34,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:35,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:16:35,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:16:35,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:35,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:36,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:37,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:38,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:39,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:40,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:40,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:41,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:42,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:43,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:43,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:16:43,407 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:16:43,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:16:43,409 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:16:44,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:44,368 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:16:45,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:45,369 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:16:45,937 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:16:45,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:46,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:47,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:48,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:49,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:50,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:16:50,164 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:16:50,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:51,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:51,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:52,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:53,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:54,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:55,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:56,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:56,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:16:57,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:58,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:59,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:16:59,379 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:17:00,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:01,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:02,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:02,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:03,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:04,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:05,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:17:05,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:17:05,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:06,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:07,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:07,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:08,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:09,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:10,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:11,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:12,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:12,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:13,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:14,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:14,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:17:14,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:17:14,880 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:17:14,882 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:17:15,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:15,399 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:17:15,941 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:17:16,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:17,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:17,401 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:17:18,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:18,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:19,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:20,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:17:20,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:17:20,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:21,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:22,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:23,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:24,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:24,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:25,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:26,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:27,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:28,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:29,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:30,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:30,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:31,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:31,412 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:17:32,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:33,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:34,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:35,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:17:35,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:17:35,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:35,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:36,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:37,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:38,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:39,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:40,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:40,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:41,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:42,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:43,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:44,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:45,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:45,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:17:45,734 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:17:45,735 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:17:45,736 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:17:45,945 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:17:45,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:46,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:46,431 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:17:47,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:47,432 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:17:48,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:49,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:50,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:17:50,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:17:50,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:51,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:51,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:52,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:53,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:54,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:55,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:56,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:56,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:17:57,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:58,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:17:59,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:00,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:01,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:01,442 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:18:01,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:02,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:03,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:04,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:05,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:18:05,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:18:05,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:06,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:07,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:07,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:08,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:09,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:10,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:11,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:12,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:12,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:13,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:14,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:15,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:15,953 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:18:16,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:17,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:17,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:18,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:18:18,146 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:18:18,146 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:18:18,148 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:18:18,459 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:18:18,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:19,460 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:18:19,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:20,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:18:20,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:18:20,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:21,462 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:18:21,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:22,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:22,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:23,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:24,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:25,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:26,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:27,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:27,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:28,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:29,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:30,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:31,468 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:18:31,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:32,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:32,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:33,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:34,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:35,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:18:35,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:18:35,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:36,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:37,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:38,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:38,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:39,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:40,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:41,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:42,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:43,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:43,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:44,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:45,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:45,960 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:18:46,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:18:46,036 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:18:46,036 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:18:46,037 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:18:46,487 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:18:46,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:47,488 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:18:47,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:48,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:49,489 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:18:49,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:49,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:50,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:18:50,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:18:50,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:51,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:52,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:53,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:54,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:54,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:18:55,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:56,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:57,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:58,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:18:59,497 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:18:59,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:00,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:00,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:01,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:02,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:03,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:04,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:05,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:19:05,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:19:05,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:06,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:06,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:07,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:08,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:09,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:10,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:11,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:11,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:12,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:19:12,436 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:19:12,436 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:19:12,438 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:19:12,515 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:19:12,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:13,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:14,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:15,517 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:19:15,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:15,960 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:19:16,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:17,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:17,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:18,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:19,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:20,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:19:20,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:19:20,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:21,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:22,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:22,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:23,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:24,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:25,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:26,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:27,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:27,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:28,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:29,526 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:19:29,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:30,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:31,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:32,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:33,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:33,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:34,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:35,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:19:35,165 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:19:35,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:36,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:37,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:38,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:38,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:39,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:40,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:41,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:42,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:43,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:19:43,325 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:19:43,325 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:19:43,327 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:19:43,545 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:19:43,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:44,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:44,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:45,547 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:19:45,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:45,964 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:19:46,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:47,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:48,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:49,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:49,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:50,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:19:50,166 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:19:50,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:51,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:52,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:53,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:54,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:19:54,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:55,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:56,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:57,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:58,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:19:59,557 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:19:59,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:00,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:00,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:01,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:02,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:03,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:04,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:05,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:20:05,166 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:20:05,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:06,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:06,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:07,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:08,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:09,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:10,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:11,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:11,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:12,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:13,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:14,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:20:14,401 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:20:14,402 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:20:14,403 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:20:14,576 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:20:14,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:15,577 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:20:15,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:15,968 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:20:16,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:16,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:17,579 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:20:17,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:18,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:19,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:20,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:20:20,166 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:20:20,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:21,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:22,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:22,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:23,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:24,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:25,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:26,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:27,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:28,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:28,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:29,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:30,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:31,589 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:20:31,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:32,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:33,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:33,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:34,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:35,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:20:35,166 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:20:35,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:36,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:37,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:38,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:39,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:39,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:40,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:41,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:42,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:43,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:44,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:44,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:45,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:45,973 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:20:46,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:46,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:20:46,889 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:20:46,889 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:20:46,891 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:20:47,609 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:20:47,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:48,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:49,610 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:20:49,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:50,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:20:50,166 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:20:50,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:50,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:51,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:52,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:53,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:54,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:55,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:20:55,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:56,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:57,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:58,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:20:59,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:00,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:01,618 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:21:01,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:01,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:02,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:03,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:04,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:05,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:21:05,167 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:21:05,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:06,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:06,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:07,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:08,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:09,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:10,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:11,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:11,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:12,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:13,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:14,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:15,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:21:15,402 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:21:15,402 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:21:15,403 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:21:15,637 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:21:15,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:15,977 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:21:16,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:16,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:17,638 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:21:17,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:18,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:19,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:20,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:21:20,166 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:21:20,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:21,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:22,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:22,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:23,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:24,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:25,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:26,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:27,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:28,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:28,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:29,647 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:21:29,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:30,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:31,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:32,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:33,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:33,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:34,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:35,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:21:35,166 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:21:35,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:36,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:37,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:38,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:39,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:39,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:40,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:41,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:42,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:43,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:44,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:44,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:45,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:45,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:21:45,951 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:21:45,951 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:21:45,952 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:21:45,979 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:21:46,666 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:21:46,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:47,667 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:21:47,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:48,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:49,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:50,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:21:50,167 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:21:50,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:50,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:51,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:52,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:53,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:54,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:55,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:21:55,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:56,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:57,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:58,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:21:59,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:00,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:01,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:01,676 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:22:01,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:02,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:03,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:04,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:05,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:22:05,167 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:22:05,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:06,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:06,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:07,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:08,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:09,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:10,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:11,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:11,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:12,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:13,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:14,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:15,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:15,990 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:22:16,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:16,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:17,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:18,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:22:18,602 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:22:18,603 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:22:18,605 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:22:18,696 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:22:19,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:20,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:20,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:22:20,167 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:22:21,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:21,699 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:22:22,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:22,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:23,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:24,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:25,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:26,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:27,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:28,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:28,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:29,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:30,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:31,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:32,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:33,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:33,707 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:22:34,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:34,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:35,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:35,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:22:35,167 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:22:36,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:37,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:38,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:39,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:39,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:40,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:41,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:42,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:43,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:44,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:45,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:45,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:45,992 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:22:46,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:47,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:48,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:49,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:50,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:50,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:22:50,169 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:22:51,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:51,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:22:51,143 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:22:51,143 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:22:51,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:51,144 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:22:51,735 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:22:52,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:53,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:53,736 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:22:54,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:55,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:56,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:56,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:22:57,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:58,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:22:59,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:00,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:01,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:02,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:02,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:03,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:04,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:05,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:05,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:23:05,169 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:23:05,745 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:23:06,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:07,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:08,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:08,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:09,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:10,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:11,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:12,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:13,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:13,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:14,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:15,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:15,991 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:23:16,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:17,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:18,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:18,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:19,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:20,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:20,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:23:20,176 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:23:21,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:22,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:22,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:23:22,218 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:23:22,218 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:23:22,223 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:23:22,766 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:23:23,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:23,766 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:23:24,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:24,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:25,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:26,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:27,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:28,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:29,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:29,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:30,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:31,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:32,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:33,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:34,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:35,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:35,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:23:35,173 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:23:35,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:36,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:37,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:37,776 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:23:38,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:39,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:40,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:40,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:41,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:42,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:43,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:44,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:45,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:45,995 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:23:45,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:46,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:47,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:48,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:49,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:50,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:50,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:23:50,176 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:23:51,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:51,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:52,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:52,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:23:52,908 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:23:52,909 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:23:52,911 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:23:53,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:53,791 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:23:54,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:55,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:55,793 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:23:56,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:56,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:23:57,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:58,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:23:59,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:00,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:01,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:01,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:02,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:03,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:04,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:05,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:24:05,176 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:24:05,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:05,800 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:24:06,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:07,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:07,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:08,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:09,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:10,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:11,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:12,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:12,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:13,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:14,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:15,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:16,012 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:24:16,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:17,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:17,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:18,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:19,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:20,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:24:20,178 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:24:20,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:21,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:21,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:24:21,990 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:24:21,990 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:24:21,992 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:24:22,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:22,824 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:24:23,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:23,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:23,825 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:24:24,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:25,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:26,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:27,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:28,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:28,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:29,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:30,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:31,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:32,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:33,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:34,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:34,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:35,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:24:35,178 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:24:35,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:35,833 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:24:36,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:37,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:38,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:39,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:39,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:40,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:41,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:42,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:43,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:44,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:45,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:45,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:46,005 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:24:46,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:47,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:48,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:49,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:50,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:24:50,189 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:24:50,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:51,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:51,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:52,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:53,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:24:53,022 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:24:53,023 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:24:53,024 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:24:53,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:53,856 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:24:54,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:55,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:55,857 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:24:56,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:56,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:24:57,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:58,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:24:59,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:00,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:01,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:01,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:02,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:03,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:03,863 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:25:04,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:05,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:25:05,184 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:25:05,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:06,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:06,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:07,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:08,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:09,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:10,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:11,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:11,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:12,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:13,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:14,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:15,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:16,007 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:25:16,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:16,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:17,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:25:17,304 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:25:17,305 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:25:17,306 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:25:17,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:17,890 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:25:18,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:19,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:19,892 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:25:20,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:25:20,184 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:25:20,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:21,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:21,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:22,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:23,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:24,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:25,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:26,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:26,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:27,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:28,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:29,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:30,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:31,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:31,900 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:25:32,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:32,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:33,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:34,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:35,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:25:35,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:25:35,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:36,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:37,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:37,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:38,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:39,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:40,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:41,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:42,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:42,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:43,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:44,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:45,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:46,012 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:25:46,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:25:46,464 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:25:46,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:25:46,465 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:25:46,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:46,918 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:25:47,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:47,919 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:25:48,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:48,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:49,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:50,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:25:50,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:25:50,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:51,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:52,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:53,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:54,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:54,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:55,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:56,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:57,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:58,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:25:59,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:25:59,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:00,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:01,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:02,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:03,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:03,930 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:26:04,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:05,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:05,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:26:05,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:26:05,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:06,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:07,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:08,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:09,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:10,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:11,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:11,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:12,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:13,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:14,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:15,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:16,015 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:26:16,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:16,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:26:16,759 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:26:16,760 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:26:16,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:16,761 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:26:16,948 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:26:17,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:17,949 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:26:18,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:19,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:19,950 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:26:20,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:26:20,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:26:20,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:21,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:22,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:22,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:23,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:24,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:25,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:26,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:27,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:27,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:28,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:29,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:30,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:31,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:31,958 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:26:32,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:32,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:33,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:34,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:35,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:26:35,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:26:35,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:36,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:37,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:38,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:38,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:39,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:40,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:41,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:42,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:43,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:43,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:44,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:26:44,510 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:26:44,511 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:26:44,516 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:26:44,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:44,986 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:26:45,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:45,987 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:26:46,018 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:26:46,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:47,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:48,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:48,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:49,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:50,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:26:50,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:26:50,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:51,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:52,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:53,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:54,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:54,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:55,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:56,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:57,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:58,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:59,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:26:59,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:26:59,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:27:00,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:01,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:02,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:03,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:04,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:05,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:05,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:27:05,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:27:05,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:06,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:07,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:08,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:09,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:10,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:11,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:11,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:12,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:13,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:14,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:15,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:16,027 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:27:16,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:27:16,649 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:27:16,649 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:27:16,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:16,650 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:27:17,019 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:27:17,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:18,020 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:27:18,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:19,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:20,022 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:27:20,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:20,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:27:20,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:27:21,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:22,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:22,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:23,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:24,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:25,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:26,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:27,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:28,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:28,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:29,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:30,028 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:27:30,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:31,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:32,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:33,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:34,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:34,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:35,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:35,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:27:35,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:27:36,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:37,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:38,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:39,105 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:39,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:40,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:41,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:42,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:43,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:43,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:27:43,473 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:27:43,474 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:27:43,476 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:27:44,049 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:27:44,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:44,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:45,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:46,024 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:27:46,051 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:27:46,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:47,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:48,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:49,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:50,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:50,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:50,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:27:50,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:27:51,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:52,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:53,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:54,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:55,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:27:55,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:56,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:57,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:58,058 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:27:58,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:27:59,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:00,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:01,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:01,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:02,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:03,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:04,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:05,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:28:05,186 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:28:05,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:06,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:06,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:07,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:08,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:09,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:10,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:11,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:12,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:12,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:12,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:28:12,526 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:28:12,526 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:28:12,527 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:28:13,076 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:28:13,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:14,077 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:28:14,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:15,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:16,028 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:28:16,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:17,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:18,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:18,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:19,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:20,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:28:20,186 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:28:20,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:21,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:22,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:23,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:24,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:24,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:25,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:26,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:27,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:28,086 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:28:28,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:29,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:29,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:30,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:31,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:32,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:33,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:34,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:34,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:35,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:28:35,186 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:28:35,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:36,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:37,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:38,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:39,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:39,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:40,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:41,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:42,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:43,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:44,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:44,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:44,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:28:44,385 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:28:44,385 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:28:44,387 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:28:45,110 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:28:45,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:46,033 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:28:46,111 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:28:46,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:47,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:48,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:49,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:50,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:50,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:28:50,186 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:28:50,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:51,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:52,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:53,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:54,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:55,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:56,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:28:56,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:57,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:58,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:28:59,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:00,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:01,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:01,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:02,122 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:29:02,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:03,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:04,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:05,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:29:05,186 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:29:05,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:06,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:07,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:07,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:08,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:09,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:10,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:11,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:12,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:13,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:13,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:14,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:15,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:16,041 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:29:16,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:17,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:18,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:18,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:18,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:29:18,728 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:29:18,728 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:29:18,730 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:29:19,143 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:29:19,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:20,144 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:29:20,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:29:20,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:29:20,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:21,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:22,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:23,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:24,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:24,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:25,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:26,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:27,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:28,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:29,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:30,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:30,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:31,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:32,152 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:29:32,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:33,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:34,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:35,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:35,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:29:35,190 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:29:35,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:36,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:37,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:38,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:39,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:40,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:40,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:41,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:42,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:43,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:44,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:45,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:45,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:46,044 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:29:46,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:47,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:48,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:29:48,041 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:29:48,041 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:29:48,043 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:29:48,171 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:29:48,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:49,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:50,173 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:29:50,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:29:50,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:29:50,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:50,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:51,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:52,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:53,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:54,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:55,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:56,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:29:56,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:57,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:58,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:29:59,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:00,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:01,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:01,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:02,182 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:30:02,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:03,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:04,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:05,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:30:05,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:30:05,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:06,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:07,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:07,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:08,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:09,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:10,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:11,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:12,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:12,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:13,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:14,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:15,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:16,046 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:30:16,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:17,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:17,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:17,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:30:17,930 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:30:17,930 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:30:17,931 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:30:18,202 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:30:18,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:19,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:20,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:30:20,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:30:20,204 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:30:20,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:21,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:22,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:22,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:23,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:24,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:25,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:26,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:27,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:28,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:28,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:29,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:30,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:31,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:32,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:33,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:33,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:34,215 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:30:34,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:35,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:30:35,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:30:35,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:36,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:37,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:38,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:39,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:39,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:40,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:41,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:42,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:43,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:44,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:44,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:45,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:46,051 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:30:46,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:47,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:48,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:49,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:49,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:30:49,727 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:30:49,727 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:30:49,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:49,729 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:30:50,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:30:50,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:30:50,233 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:30:50,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:51,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:52,235 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:30:52,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:53,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:54,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:55,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:30:55,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:56,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:57,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:58,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:30:59,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:00,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:01,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:01,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:02,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:03,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:04,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:05,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:31:05,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:31:05,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:06,245 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:31:06,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:06,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:07,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:08,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:09,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:10,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:11,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:12,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:12,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:13,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:14,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:15,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:16,058 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:31:16,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:17,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:17,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:18,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:19,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:19,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:31:19,640 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:31:19,640 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:31:19,642 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:31:20,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:31:20,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:31:20,269 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:31:20,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:21,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:22,270 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:31:22,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:23,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:23,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:24,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:25,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:26,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:27,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:28,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:28,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:29,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:30,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:31,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:32,279 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:31:32,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:33,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:34,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:34,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:35,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:31:35,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:31:35,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:36,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:37,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:38,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:39,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:40,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:40,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:41,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:42,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:43,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:44,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:45,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:45,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:46,058 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:31:46,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:47,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:48,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:48,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:31:48,607 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:31:48,607 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:31:48,609 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:31:49,308 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:31:49,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:50,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:31:50,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:31:50,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:50,309 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:31:50,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:51,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:52,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:53,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:54,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:55,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:56,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:31:56,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:57,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:58,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:31:59,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:00,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:01,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:01,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:02,319 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:32:02,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:03,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:04,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:05,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:32:05,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:32:05,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:06,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:06,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:07,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:08,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:09,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:10,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:11,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:12,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:12,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:13,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:14,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:15,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:16,062 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:32:16,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:17,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:17,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:17,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:32:17,657 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:32:17,658 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:32:17,661 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:32:18,344 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:32:18,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:19,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:20,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:32:20,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:32:20,346 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:32:20,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:21,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:22,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:22,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:23,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:24,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:25,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:26,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:27,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:27,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:28,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:29,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:30,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:31,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:32,354 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:32:32,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:33,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:33,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:34,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:35,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:32:35,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:32:35,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:36,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:37,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:38,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:38,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:39,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:40,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:41,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:42,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:43,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:43,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:44,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:45,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:46,063 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:32:46,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:46,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:32:46,678 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:32:46,679 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:32:46,680 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:32:47,372 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:32:47,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:48,373 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:32:48,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:49,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:49,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:50,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:32:50,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:32:50,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:51,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:52,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:53,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:54,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:32:54,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:55,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:56,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:57,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:58,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:32:59,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:00,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:00,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:01,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:02,383 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:33:02,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:03,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:04,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:05,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:33:05,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:33:05,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:05,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:06,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:07,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:08,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:09,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:10,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:11,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:11,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:12,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:13,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:14,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:15,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:16,073 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:33:16,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:17,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:17,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:18,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:33:18,112 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:33:18,113 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:33:18,117 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:33:18,403 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:33:18,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:19,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:20,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:33:20,193 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:33:20,405 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:33:20,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:21,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:22,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:22,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:23,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:24,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:25,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:26,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:27,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:28,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:28,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:29,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:30,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:31,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:32,414 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:33:32,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:33,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:33,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:34,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:35,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:33:35,192 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:33:35,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:36,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:37,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:38,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:39,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:39,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:40,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:41,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:42,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:43,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:44,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:44,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:45,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:46,080 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:33:46,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:47,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:48,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:49,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:49,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:33:49,764 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:33:49,764 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:33:49,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:49,770 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:33:50,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:33:50,193 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:33:50,434 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:33:50,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:51,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:52,436 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:33:52,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:53,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:54,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:55,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:33:55,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:56,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:57,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:58,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:33:59,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:00,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:01,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:01,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:02,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:03,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:04,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:05,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:34:05,193 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:34:05,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:06,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:06,446 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:34:06,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:07,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:08,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:09,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:10,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:11,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:12,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:12,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:13,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:14,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:15,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:16,078 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:34:16,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:17,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:17,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:18,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:19,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:20,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:34:20,177 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:34:20,177 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:34:20,182 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:34:20,468 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:34:20,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:34:20,588 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:34:20,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:21,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:22,470 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:34:22,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:23,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:23,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:24,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:25,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:26,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:27,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:28,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:29,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:29,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:30,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:31,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:32,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:33,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:34,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:34,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:35,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:34:35,318 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:34:35,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:36,479 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:34:36,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:37,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:38,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:39,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:40,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:40,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:41,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:42,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:43,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:44,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:45,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:46,079 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:34:46,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:46,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:47,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:48,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:49,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:50,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:34:50,324 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:34:50,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:51,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:51,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:52,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:53,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:34:53,035 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:34:53,036 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:34:53,038 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:34:53,505 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:34:53,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:54,506 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:34:54,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:55,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:56,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:34:56,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:57,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:58,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:34:59,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:00,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:01,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:02,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:02,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:03,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:04,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:05,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:35:05,324 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:35:05,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:06,515 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:35:06,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:07,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:08,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:08,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:09,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:10,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:11,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:12,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:13,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:14,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:14,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:15,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:16,083 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:35:16,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:17,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:18,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:19,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:19,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:20,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:35:20,291 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:35:20,291 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:35:20,292 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:35:20,533 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:35:20,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:35:20,697 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:35:20,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:21,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:22,535 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:35:22,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:23,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:24,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:25,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:25,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:26,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:27,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:28,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:29,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:30,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:31,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:31,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:32,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:33,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:34,543 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:35:34,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:35,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:35:35,431 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:35:35,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:36,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:36,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:37,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:38,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:39,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:40,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:41,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:42,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:42,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:43,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:44,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:45,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:46,088 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:35:46,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:47,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:47,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:48,709 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:49,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:50,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:35:50,042 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:35:50,042 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:35:50,044 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:35:50,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:35:50,458 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:35:50,570 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:35:50,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:51,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:52,571 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:35:52,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:53,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:53,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:54,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:55,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:56,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:57,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:58,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:35:58,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:35:59,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:00,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:01,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:02,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:03,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:03,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:04,580 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:36:04,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:05,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:36:05,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:36:05,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:06,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:07,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:08,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:09,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:09,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:10,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:11,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:12,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:13,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:14,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:14,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:15,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:16,089 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:36:16,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:17,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:18,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:36:18,458 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:36:18,459 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:36:18,460 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:36:18,602 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:36:18,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:19,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:19,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:20,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:36:20,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:36:20,604 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:36:20,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:21,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:22,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:23,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:24,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:25,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:25,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:26,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:27,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:28,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:29,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:30,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:31,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:31,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:32,612 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:36:32,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:33,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:34,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:35,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:36:35,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:36:35,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:36,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:36,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:37,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:38,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:39,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:40,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:41,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:42,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:42,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:43,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:44,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:45,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:46,102 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:36:46,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:47,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:47,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:48,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:36:48,835 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:36:48,836 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:36:48,837 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:36:49,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:49,639 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:36:49,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:50,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:36:50,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:36:50,640 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:36:50,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:51,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:52,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:53,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:53,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:54,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:55,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:56,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:57,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:58,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:36:58,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:36:59,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:00,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:01,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:02,649 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:37:02,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:03,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:03,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:04,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:05,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:37:05,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:37:05,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:06,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:07,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:08,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:08,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:09,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:10,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:11,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:12,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:13,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:13,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:14,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:15,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:16,099 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:37:16,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:37:16,444 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:37:16,445 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:37:16,450 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:37:16,678 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:37:16,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:17,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:18,679 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:37:18,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:19,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:19,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:20,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:37:20,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:37:20,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:21,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:22,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:23,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:24,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:25,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:25,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:26,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:27,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:28,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:30,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:31,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:31,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:32,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:32,691 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:37:33,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:34,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:35,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:35,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:37:35,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:37:36,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:36,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:37,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:38,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:39,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:40,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:41,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:42,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:42,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:43,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:44,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:45,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:46,103 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:37:46,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:47,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:47,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:48,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:48,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:37:48,340 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:37:48,341 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:37:48,346 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:37:48,709 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:37:49,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:50,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:50,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:37:50,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:37:50,711 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:37:51,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:52,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:53,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:53,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:37:54,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:55,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:56,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:57,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:58,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:59,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:37:59,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:00,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:01,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:02,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:02,719 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:38:03,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:04,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:04,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:05,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:05,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:38:05,451 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:38:06,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:07,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:08,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:09,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:10,140 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:10,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:11,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:12,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:13,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:14,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:15,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:16,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:38:16,057 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:38:16,058 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:38:16,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:16,059 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:38:16,107 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:38:16,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:16,738 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:38:17,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:18,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:18,740 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:38:19,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:20,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:38:20,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:38:20,491 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:21,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:21,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:22,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:23,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:24,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:25,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:26,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:27,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:27,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:28,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:29,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:30,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:30,749 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:38:31,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:32,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:32,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:33,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:34,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:35,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:38:35,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:38:35,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:36,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:37,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:38,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:38,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:39,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:40,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:41,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:42,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:43,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:43,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:44,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:45,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:46,112 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:38:46,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:47,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:47,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:38:47,787 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:38:47,787 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:38:47,789 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:38:48,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:48,767 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:38:48,768 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:38:49,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:49,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:50,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:38:50,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:38:50,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:50,769 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:38:51,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:52,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:53,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:54,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:54,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:55,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:56,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:57,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:58,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:38:59,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:38:59,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:00,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:01,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:02,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:03,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:04,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:04,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:04,777 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:39:05,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:39:05,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:39:05,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:06,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:07,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:08,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:09,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:10,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:10,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:11,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:12,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:13,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:14,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:15,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:15,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:16,118 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:39:16,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:17,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:17,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:39:17,822 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:39:17,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:39:17,825 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:39:18,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:18,802 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:39:18,803 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:39:19,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:20,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:39:20,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:39:20,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:20,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:20,804 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:39:21,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:22,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:23,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:24,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:25,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:26,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:26,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:27,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:28,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:29,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:30,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:31,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:31,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:32,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:32,812 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:39:33,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:34,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:35,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:39:35,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:39:35,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:36,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:36,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:37,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:38,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:39,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:40,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:41,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:42,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:42,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:43,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:44,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:45,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:46,120 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:39:46,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:47,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:47,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:48,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:39:48,434 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:39:48,434 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:39:48,436 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:39:48,833 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:39:48,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:49,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:50,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:39:50,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:39:50,835 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:39:50,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:51,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:52,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:53,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:53,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:54,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:55,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:56,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:57,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:58,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:39:59,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:39:59,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:00,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:01,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:02,843 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:40:02,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:03,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:04,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:05,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:05,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:40:05,452 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:40:05,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:06,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:07,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:08,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:09,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:10,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:11,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:11,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:12,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:13,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:14,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:15,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:16,127 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:40:16,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:17,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:17,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:18,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:40:18,497 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:40:18,497 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:40:18,498 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:40:18,867 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:40:18,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:19,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:20,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:40:20,453 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:40:20,869 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:40:20,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:21,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:22,250 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:22,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:23,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:24,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:25,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:26,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:27,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:28,250 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:28,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:29,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:30,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:31,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:32,877 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:40:32,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:33,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:33,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:34,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:35,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:40:35,453 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:40:35,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:36,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:37,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:38,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:39,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:39,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:40,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:41,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:42,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:43,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:44,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:45,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:45,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:46,128 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:40:46,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:47,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:48,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:49,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:50,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:50,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:40:50,406 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:40:50,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:40:50,410 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:40:50,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:40:50,813 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:40:50,903 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:40:50,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:51,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:52,904 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:40:52,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:53,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:54,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:55,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:56,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:40:56,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:57,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:58,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:40:59,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:00,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:01,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:01,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:03,150 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:04,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:04,915 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:41:05,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:05,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:41:05,555 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:41:06,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:06,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:07,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:08,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:09,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:10,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:11,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:12,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:12,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:13,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:14,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:15,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:16,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:16,141 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:41:17,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:17,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:18,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:19,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:20,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:20,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:41:20,555 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:41:21,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:21,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:41:21,772 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:41:21,772 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:41:21,775 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:41:21,932 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:41:22,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:22,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:22,933 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:41:23,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:24,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:25,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:26,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:27,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:28,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:28,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:29,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:30,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:31,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:32,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:33,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:33,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:34,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:34,941 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:41:35,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:35,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:41:35,555 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:41:36,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:37,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:38,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:38,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:39,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:40,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:41,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:42,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:43,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:44,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:44,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:45,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:46,138 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:41:46,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:47,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:48,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:49,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:49,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:50,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:50,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:41:50,556 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:41:50,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:41:50,922 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:41:50,923 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:41:50,924 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:41:50,960 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:41:51,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:52,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:52,962 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:41:53,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:54,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:55,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:41:55,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:56,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:57,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:58,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:41:59,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:00,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:00,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:01,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:02,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:03,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:04,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:04,971 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:42:05,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:05,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:42:05,556 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:42:05,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:06,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:07,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:08,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:09,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:10,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:11,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:11,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:12,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:13,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:14,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:15,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:16,139 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:42:16,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:17,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:17,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:18,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:19,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:19,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:42:19,740 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:42:19,740 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:42:19,742 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:42:19,994 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:42:20,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:20,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:42:20,556 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:42:20,995 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:42:21,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:22,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:22,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:23,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:24,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:25,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:26,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:27,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:27,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:28,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:29,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:30,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:31,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:32,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:33,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:33,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:34,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:35,006 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:42:35,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:35,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:42:35,557 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:42:36,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:37,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:38,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:38,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:39,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:40,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:41,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:42,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:43,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:44,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:44,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:45,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:46,157 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:42:46,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:47,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:48,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:49,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:49,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:50,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:50,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:42:50,560 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:42:51,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:51,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:42:51,507 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:42:51,507 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:42:51,509 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:42:52,026 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:42:52,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:53,027 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:42:53,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:54,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:55,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:42:55,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:56,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:57,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:58,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:42:59,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:00,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:01,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:01,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:02,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:03,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:04,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:05,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:05,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:43:05,560 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:43:06,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:06,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:07,036 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:43:07,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:08,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:09,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:10,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:11,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:12,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:12,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:13,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:14,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:15,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:16,149 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:43:16,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:17,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:17,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:18,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:19,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:20,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:20,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:43:20,564 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:43:21,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:22,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:22,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:43:22,602 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:43:22,602 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:43:22,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:22,603 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:43:23,066 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:43:23,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:24,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:25,067 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:43:25,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:26,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:27,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:28,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:28,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:29,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:30,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:31,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:32,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:33,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:34,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:34,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:35,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:35,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:43:35,564 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:43:36,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:37,076 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:43:37,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:38,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:39,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:39,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:40,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:41,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:42,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:43,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:44,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:45,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:45,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:46,153 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:43:46,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:47,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:48,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:49,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:50,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:50,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:50,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:43:50,566 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:43:51,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:52,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:53,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:54,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:55,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:43:55,041 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:43:55,041 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:43:55,042 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:43:55,095 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:43:55,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:55,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:43:56,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:57,096 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:43:57,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:58,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:43:59,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:00,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:01,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:01,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:02,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:03,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:04,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:05,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:05,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:44:05,566 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:44:06,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:06,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:07,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:08,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:09,105 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:44:09,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:10,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:11,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:11,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:12,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:13,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:14,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:15,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:16,158 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:44:16,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:17,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:17,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:18,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:19,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:20,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:20,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:44:20,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:44:21,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:22,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:22,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:23,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:24,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:25,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:44:25,362 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:44:25,363 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:44:25,364 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:44:25,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:26,129 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:44:26,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:27,130 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:44:27,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:27,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:28,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:29,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:30,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:31,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:32,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:32,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:33,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:34,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:35,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:35,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:44:35,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:44:36,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:37,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:38,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:38,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:39,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:40,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:41,141 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:44:41,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:42,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:43,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:43,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:44,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:45,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:46,161 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:44:46,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:47,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:48,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:49,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:49,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:50,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:50,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:44:50,574 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:44:51,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:52,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:53,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:54,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:44:54,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:55,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:56,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:44:56,406 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:44:56,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:44:56,407 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:44:56,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:57,163 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:44:57,164 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:44:57,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:58,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:44:59,165 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:44:59,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:00,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:00,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:01,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:02,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:03,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:04,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:05,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:45:05,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:45:05,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:05,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:06,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:07,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:08,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:09,173 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:45:09,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:10,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:10,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:11,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:12,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:13,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:14,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:15,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:16,165 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:45:16,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:16,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:17,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:18,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:19,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:20,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:45:20,571 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:45:20,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:21,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:21,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:22,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:23,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:45:23,648 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:45:23,649 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:45:23,650 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:45:24,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:24,194 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:45:25,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:25,195 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:45:26,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:27,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:27,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:28,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:29,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:30,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:31,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:32,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:32,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:33,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:34,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:35,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:35,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:45:35,571 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:45:36,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:37,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:37,203 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:45:37,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:38,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:39,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:40,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:41,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:42,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:43,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:43,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:44,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:45,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:46,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:46,169 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:45:47,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:48,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:48,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:49,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:50,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:50,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:45:50,574 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:45:51,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:52,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:53,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:54,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:45:54,038 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:45:54,038 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:45:54,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:45:54,040 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:45:54,222 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:45:54,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:55,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:55,223 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:45:56,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:57,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:58,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:59,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:45:59,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:00,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:01,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:02,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:03,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:04,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:04,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:05,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:05,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:46:05,574 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:46:06,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:07,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:08,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:09,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:09,231 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:46:09,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:10,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:11,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:12,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:13,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:14,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:15,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:15,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:16,173 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:46:16,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:17,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:18,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:19,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:20,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:20,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:20,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:46:20,583 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:46:21,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:22,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:23,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:24,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:25,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:46:25,147 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:46:25,148 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:46:25,149 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:46:25,251 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:46:25,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:25,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:26,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:27,253 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:46:27,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:28,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:29,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:30,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:31,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:31,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:32,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:33,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:34,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:35,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:35,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:46:35,576 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:46:36,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:36,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:37,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:38,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:39,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:40,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:41,263 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:46:41,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:41,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:42,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:43,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:44,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:45,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:46,178 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:46:46,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:47,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:47,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:48,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:49,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:50,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:50,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:46:50,576 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:46:51,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:52,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:52,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:53,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:54,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:55,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:56,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:57,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:46:57,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:58,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:46:58,202 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:46:58,202 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:46:58,203 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:46:58,289 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:46:59,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:46:59,290 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:46:59,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:00,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:01,292 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:47:01,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:02,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:02,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:03,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:04,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:05,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:05,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:47:05,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:47:06,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:07,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:08,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:08,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:09,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:10,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:11,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:12,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:13,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:13,300 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:47:13,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:14,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:15,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:16,182 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:47:16,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:17,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:18,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:19,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:19,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:20,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:20,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:47:20,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:47:21,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:22,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:23,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:24,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:24,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:25,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:26,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:27,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:27,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:47:27,816 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:47:27,816 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:47:27,818 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:47:28,322 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:47:28,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:29,323 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:47:29,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:30,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:30,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:31,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:32,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:33,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:34,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:35,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:35,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:35,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:47:35,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:47:36,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:37,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:38,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:39,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:40,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:41,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:41,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:42,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:43,333 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:47:43,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:44,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:45,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:46,186 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:47:46,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:47,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:47,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:48,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:49,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:50,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:50,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:47:50,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:47:51,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:52,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:52,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:53,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:54,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:55,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:56,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:57,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:47:57,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:58,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:59,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:47:59,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:47:59,634 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:47:59,635 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:47:59,636 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:48:00,351 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:48:00,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:01,353 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:48:01,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:02,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:02,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:03,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:04,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:05,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:05,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:48:05,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:48:06,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:07,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:08,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:08,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:09,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:10,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:11,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:12,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:13,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:14,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:14,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:15,362 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:48:15,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:16,190 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:48:16,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:17,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:18,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:19,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:19,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:20,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:20,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:48:20,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:48:21,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:22,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:23,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:24,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:25,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:25,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:26,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:27,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:28,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:29,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:30,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:30,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:30,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:48:30,778 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:48:30,778 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:48:30,779 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:48:31,389 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:48:31,390 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:48:31,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:32,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:33,391 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:48:33,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:34,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:35,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:35,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:48:35,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:48:35,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:36,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:37,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:38,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:39,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:40,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:41,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:41,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:42,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:43,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:44,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:45,399 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:48:45,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:46,193 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:48:46,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:47,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:47,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:48,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:49,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:50,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:50,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:48:50,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:48:51,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:52,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:52,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:53,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:54,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:55,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:56,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:57,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:48:57,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:58,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:48:59,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:00,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:00,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:49:00,908 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:49:00,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:49:00,910 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:49:01,420 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:49:01,420 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:49:01,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:02,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:03,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:03,422 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:49:03,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:04,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:05,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:05,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:49:05,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:49:06,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:07,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:08,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:09,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:09,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:10,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:11,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:12,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:13,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:14,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:14,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:15,429 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:49:15,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:16,197 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:49:16,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:17,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:18,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:19,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:20,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:20,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:20,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:49:20,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:49:21,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:22,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:23,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:24,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:25,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:25,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:26,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:27,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:28,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:29,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:30,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:30,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:31,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:49:31,014 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:49:31,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:49:31,016 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:49:31,445 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:49:31,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:32,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:33,448 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:49:33,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:34,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:35,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:49:35,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:49:35,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:35,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:36,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:37,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:38,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:39,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:40,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:41,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:41,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:42,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:43,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:44,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:45,455 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:49:45,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:46,201 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:49:46,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:47,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:47,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:48,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:49,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:50,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:49:50,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:49:50,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:51,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:52,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:52,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:53,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:54,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:55,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:56,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:57,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:49:57,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:58,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:49:59,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:00,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:00,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:50:00,896 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:50:00,897 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:50:00,898 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:50:01,479 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:50:01,479 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:50:01,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:02,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:02,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:03,480 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:50:03,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:04,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:05,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:50:05,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:50:05,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:06,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:07,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:08,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:08,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:09,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:10,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:11,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:12,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:13,487 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:50:13,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:13,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:14,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:15,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:16,205 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:50:16,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:17,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:18,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:19,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:19,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:20,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:50:20,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:50:20,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:21,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:22,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:23,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:24,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:24,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:25,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:26,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:27,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:28,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:50:28,148 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:50:28,149 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:50:28,150 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:50:28,509 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:50:29,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:29,510 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:50:30,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:30,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:31,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:31,511 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:50:32,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:33,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:34,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:35,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:35,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:35,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:50:35,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:50:36,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:37,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:38,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:39,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:40,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:40,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:41,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:42,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:43,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:43,519 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:50:44,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:45,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:45,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:46,209 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:50:46,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:47,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:48,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:49,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:50,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:50,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:50:50,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:50:50,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:51,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:52,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:53,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:54,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:55,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:56,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:56,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:50:57,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:58,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:50:59,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:00,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:51:00,142 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:51:00,143 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:51:00,144 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:51:00,540 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:51:00,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:01,541 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:51:01,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:01,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:02,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:03,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:04,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:05,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:51:05,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:51:05,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:06,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:06,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:07,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:08,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:09,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:10,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:11,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:11,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:12,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:13,547 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:51:13,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:14,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:15,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:16,213 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:51:16,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:17,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:17,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:18,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:19,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:20,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:51:20,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:51:20,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:21,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:22,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:22,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:23,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:24,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:25,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:26,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:27,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:27,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:28,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:29,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:51:29,182 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:51:29,182 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:51:29,183 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:51:29,563 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:51:30,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:31,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:31,565 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:51:32,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:32,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:33,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:34,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:35,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:35,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:51:35,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:51:36,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:37,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:38,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:38,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:39,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:40,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:41,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:42,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:43,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:44,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:44,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:45,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:45,575 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:51:46,217 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:51:46,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:47,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:48,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:49,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:49,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:50,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:50,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:51:50,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:51:51,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:52,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:53,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:54,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:54,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:55,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:56,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:57,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:58,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:51:59,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:51:59,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:00,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:01,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:02,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:52:02,320 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:52:02,320 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:52:02,322 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:52:02,591 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:52:02,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:03,592 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:52:03,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:04,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:04,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:05,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:52:05,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:52:05,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:06,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:07,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:08,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:09,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:10,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:10,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:11,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:12,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:13,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:14,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:15,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:16,222 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:52:16,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:16,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:17,602 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:52:17,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:18,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:19,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:20,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:52:20,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:52:20,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:21,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:21,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:22,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:23,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:24,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:25,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:26,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:27,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:27,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:28,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:29,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:30,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:31,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:32,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:32,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:33,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:34,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:52:34,111 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:52:34,112 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:52:34,113 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:52:34,626 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:52:34,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:35,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:52:35,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:52:35,626 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:52:35,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:36,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:37,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:37,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:38,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:39,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:40,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:41,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:42,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:43,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:43,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:44,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:45,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:46,226 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:52:46,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:47,634 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:52:47,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:48,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:48,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:49,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:50,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:52:50,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:52:50,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:51,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:52,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:53,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:54,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:54,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:55,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:56,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:57,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:58,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:52:59,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:52:59,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:00,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:01,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:02,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:03,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:04,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:53:04,104 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:53:04,105 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:53:04,106 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:53:04,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:04,660 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:53:04,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:05,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:53:05,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:53:05,661 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:53:05,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:06,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:07,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:08,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:09,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:10,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:10,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:11,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:12,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:13,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:14,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:15,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:15,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:16,230 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:53:16,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:17,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:18,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:19,669 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:53:19,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:20,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:53:20,580 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:53:20,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:20,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:21,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:22,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:23,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:24,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:25,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:26,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:26,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:27,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:28,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:29,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:30,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:31,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:31,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:32,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:33,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:34,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:35,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:53:35,580 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:53:35,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:53:35,715 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:53:35,715 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:53:35,718 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:53:36,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:36,688 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:53:36,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:37,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:37,689 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:53:37,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:38,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:39,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:40,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:41,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:42,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:42,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:43,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:44,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:45,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:46,234 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:53:46,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:47,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:47,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:48,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:49,697 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:53:49,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:50,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:53:50,580 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:53:50,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:51,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:52,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:53,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:53,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:54,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:55,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:56,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:57,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:58,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:53:58,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:53:59,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:00,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:01,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:02,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:03,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:03,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:04,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:54:04,576 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:54:04,576 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:54:04,578 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:54:04,719 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:54:04,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:05,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:54:05,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:54:05,720 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:54:05,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:06,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:07,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:08,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:09,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:09,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:10,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:11,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:12,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:13,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:14,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:15,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:15,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:16,238 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:54:16,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:17,729 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:54:17,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:18,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:19,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:20,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:54:20,581 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:54:20,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:20,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:21,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:22,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:23,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:24,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:25,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:26,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:26,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:27,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:28,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:29,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:30,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:31,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:31,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:32,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:33,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:34,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:54:34,152 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:54:34,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:54:34,154 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:54:34,749 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:54:34,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:35,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:54:35,581 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:54:35,750 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:54:35,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:36,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:36,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:37,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:38,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:39,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:40,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:41,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:42,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:42,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:43,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:44,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:45,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:46,242 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:54:46,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:47,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:47,757 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:54:47,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:48,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:49,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:50,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:54:50,581 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:54:50,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:51,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:52,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:52,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:53,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:54,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:55,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:56,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:57,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:54:57,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:58,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:54:59,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:00,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:01,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:02,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:02,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:03,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:55:03,113 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:55:03,114 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:55:03,115 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:55:03,781 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:55:03,782 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:55:03,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:04,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:05,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:55:05,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:55:05,784 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:55:05,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:06,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:07,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:07,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:08,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:09,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:10,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:11,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:12,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:12,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:13,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:14,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:15,791 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:55:15,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:16,246 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:55:16,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:17,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:17,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:18,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:19,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:20,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:55:20,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:55:20,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:21,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:22,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:23,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:23,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:24,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:25,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:26,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:28,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:28,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:29,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:30,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:31,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:32,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:33,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:33,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:55:33,206 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:55:33,206 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:55:33,207 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:55:33,811 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:55:33,812 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:55:34,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:34,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:35,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:35,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:55:35,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:55:35,813 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:55:36,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:37,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:38,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:39,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:39,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:40,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:41,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:42,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:43,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:44,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:45,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:45,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:46,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:46,250 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:55:47,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:47,822 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:55:48,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:49,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:50,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:50,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:55:50,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:55:50,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:51,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:52,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:53,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:54,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:55,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:56,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:56,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:55:57,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:58,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:55:59,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:00,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:01,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:01,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:02,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:03,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:03,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:56:03,200 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:56:03,200 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:56:03,202 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:56:03,841 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:56:03,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:56:04,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:05,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:05,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:56:05,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:56:05,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:56:06,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:06,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:07,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:08,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:09,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:10,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:11,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:12,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:12,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:13,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:14,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:15,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:16,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:16,254 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:56:17,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:17,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:17,850 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:56:18,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:19,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:20,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:20,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:56:20,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:56:21,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:22,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:23,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:23,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:24,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:25,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:26,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:27,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:28,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:28,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:29,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:30,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:31,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:32,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:33,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:33,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:56:33,625 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:56:33,625 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:56:33,627 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:56:33,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:33,876 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:56:34,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:35,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:35,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:56:35,582 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:56:35,877 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:56:36,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:37,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:38,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:39,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:39,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:40,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:41,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:42,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:43,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:44,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:45,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:45,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:46,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:46,258 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:56:47,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:48,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:49,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:49,887 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:56:50,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:50,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:56:50,583 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:56:50,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:51,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:52,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:53,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:54,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:55,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:56,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:56,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:56:57,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:58,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:56:59,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:00,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:01,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:02,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:02,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:03,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:04,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:05,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:05,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:57:05,589 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:57:06,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:07,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:07,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:57:07,711 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:57:07,712 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:57:07,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:07,714 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:57:07,905 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:57:08,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:09,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:09,907 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:57:10,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:11,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:12,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:13,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:13,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:14,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:15,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:16,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:16,263 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:57:17,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:18,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:19,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:19,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:20,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:20,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:57:20,586 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:57:21,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:21,915 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:57:22,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:23,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:24,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:25,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:25,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:26,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:27,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:28,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:29,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:30,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:30,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:31,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:32,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:33,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:34,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:35,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:35,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:57:35,590 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:57:35,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:36,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:37,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:38,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:38,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:57:38,887 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:57:38,887 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:57:38,888 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:57:38,931 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:57:39,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:39,932 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:57:40,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:41,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:41,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:42,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:43,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:44,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:45,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:46,266 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:57:46,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:47,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:47,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:48,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:49,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:50,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:50,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:57:50,590 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:57:51,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:52,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:52,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:53,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:53,942 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:57:54,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:55,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:56,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:57,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:57,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:57:58,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:57:59,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:00,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:01,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:02,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:03,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:03,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:04,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:05,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:05,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:58:05,607 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:58:06,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:07,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:08,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:08,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:09,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:09,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:58:09,930 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:58:09,930 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:58:09,932 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:58:09,962 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:58:10,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:11,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:11,963 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:58:12,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:13,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:14,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:14,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:15,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:16,270 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:58:16,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:17,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:18,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:19,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:20,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:20,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:58:20,594 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:58:20,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:21,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:21,970 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:58:22,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:23,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:24,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:25,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:26,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:26,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:27,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:28,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:29,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:30,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:31,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:32,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:32,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:33,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:34,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:35,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:35,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:58:35,602 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:58:36,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:37,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:37,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:38,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:38,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:58:38,970 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:58:38,971 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:58:38,972 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:58:38,995 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:58:39,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:39,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:58:40,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:41,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:42,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:42,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:43,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:44,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:45,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:46,274 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:58:46,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:47,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:48,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:48,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:49,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:50,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:50,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:58:50,602 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:58:51,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:52,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:53,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:53,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:58:54,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:55,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:56,007 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:58:56,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:57,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:58,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:59,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:58:59,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:00,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:01,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:02,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:03,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:04,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:05,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:05,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:59:05,602 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:59:05,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:06,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:07,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:08,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:09,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:10,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:11,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:11,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:12,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:12,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:59:12,588 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:59:12,589 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:59:12,590 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:59:13,029 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:59:13,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:14,030 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:59:14,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:15,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:16,278 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:59:16,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:17,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:17,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:18,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:19,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:20,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:20,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:59:20,603 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:59:21,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:22,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:22,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:23,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:24,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:25,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:26,039 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:59:26,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:27,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:27,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:28,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:29,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:30,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:31,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:32,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:32,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:33,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:34,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:35,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:35,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:59:35,606 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:59:36,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:37,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:37,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:38,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:39,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:40,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 10:59:40,346 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 10:59:40,346 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 10:59:40,348 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 10:59:40,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:41,060 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 10:59:41,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:42,060 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:59:42,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:42,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:43,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:44,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:45,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:46,282 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 10:59:46,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:47,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:48,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:48,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:49,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:50,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 10:59:50,603 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 10:59:50,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:51,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:52,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:53,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:54,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:54,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:55,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:56,070 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 10:59:56,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:57,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:58,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 10:59:59,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 10:59:59,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:00,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:01,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:02,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:03,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:04,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:05,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:00:05,603 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:00:05,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:05,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:06,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:07,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:08,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:09,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:10,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:10,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:11,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:12,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:12,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:00:12,978 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:00:12,979 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:00:12,980 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:00:13,089 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:00:13,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:14,091 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:00:14,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:15,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:16,286 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:00:16,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:16,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:17,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:18,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:19,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:20,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:00:20,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:00:20,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:21,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:21,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:22,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:23,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:24,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:25,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:26,099 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:00:26,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:27,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:27,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:28,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:29,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:30,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:31,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:32,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:32,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:33,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:34,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:35,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:00:35,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:00:35,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:36,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:37,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:37,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:38,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:39,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:40,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:41,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:42,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:43,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:00:43,500 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:00:43,501 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:00:43,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:43,503 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:00:43,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:44,129 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:00:44,130 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:00:44,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:45,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:46,289 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:00:46,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:47,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:48,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:48,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:49,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:50,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:00:50,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:00:50,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:51,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:52,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:53,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:54,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:54,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:55,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:56,137 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:00:56,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:57,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:58,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:00:59,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:00:59,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:00,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:01,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:02,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:03,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:04,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:05,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:01:05,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:01:05,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:05,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:06,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:07,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:08,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:09,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:10,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:10,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:11,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:12,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:13,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:01:13,259 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:01:13,259 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:01:13,261 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:01:13,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:14,155 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:01:14,155 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:01:14,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:15,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:16,294 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:01:16,300 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:16,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:17,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:18,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:19,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:20,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:01:20,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:01:20,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:21,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:21,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:22,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:23,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:24,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:25,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:26,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:27,487 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:27,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:28,164 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:01:28,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:29,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:30,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:31,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:32,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:32,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:33,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:34,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:35,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:01:35,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:01:35,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:36,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:37,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:37,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:38,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:39,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:40,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:41,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:42,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:42,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:43,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:43,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:01:43,944 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:01:43,945 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:01:43,947 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:01:44,187 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:01:44,187 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:01:44,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:45,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:46,188 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:01:46,298 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:01:46,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:47,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:48,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:48,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:49,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:50,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:01:50,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:01:50,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:51,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:52,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:53,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:54,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:54,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:55,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:56,195 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:01:56,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:57,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:58,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:01:59,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:01:59,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:00,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:01,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:02,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:03,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:04,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:04,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:05,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:02:05,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:02:05,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:06,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:07,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:08,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:09,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:09,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:10,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:11,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:12,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:02:12,525 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:02:12,525 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:02:12,526 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:02:12,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:13,218 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:02:13,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:14,219 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:02:14,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:14,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:15,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:16,302 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:02:16,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:17,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:18,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:19,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:20,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:02:20,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:02:20,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:20,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:21,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:22,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:23,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:24,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:25,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:25,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:27,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:28,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:28,229 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:02:29,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:30,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:30,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:31,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:32,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:33,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:34,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:35,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:35,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:02:35,605 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:02:36,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:36,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:37,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:38,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:39,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:40,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:41,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:41,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:42,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:43,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:44,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:45,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:45,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:02:45,233 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:02:45,234 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:02:45,236 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:02:45,257 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:02:46,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:46,258 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:02:46,306 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:02:47,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:47,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:48,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:49,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:50,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:50,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:02:50,605 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:02:51,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:52,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:52,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:53,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:54,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:55,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:56,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:57,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:57,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:02:58,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:02:58,266 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:02:59,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:00,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:01,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:02,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:02,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:03,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:04,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:05,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:05,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:03:05,605 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:03:06,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:07,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:08,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:08,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:09,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:10,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:11,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:12,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:13,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:14,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:14,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:03:14,304 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:03:14,304 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:03:14,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:14,306 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:03:15,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:15,289 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:03:16,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:16,290 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:03:16,310 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:03:17,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:18,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:19,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:19,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:20,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:20,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:03:20,605 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:03:21,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:22,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:23,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:24,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:25,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:25,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:26,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:26,297 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:03:27,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:28,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:29,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:30,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:30,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:31,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:32,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:33,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:34,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:35,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:35,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:03:35,619 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:03:36,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:36,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:37,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:38,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:39,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:40,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:41,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:41,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:03:41,113 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:03:41,113 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:03:41,115 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:03:41,314 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:03:42,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:42,315 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:03:42,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:43,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:44,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:45,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:46,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:46,314 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:03:47,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:47,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:48,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:49,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:50,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:50,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:03:50,611 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:03:51,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:52,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:52,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:53,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:54,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:54,326 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:03:55,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:56,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:57,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:57,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:03:58,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:03:59,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:00,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:01,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:02,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:03,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:03,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:04,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:05,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:05,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:04:05,615 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:04:06,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:07,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:08,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:08,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:09,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:10,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:04:10,010 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:04:10,010 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:04:10,012 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:04:10,341 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:04:10,342 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:04:10,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:11,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:12,343 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:04:12,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:13,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:14,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:14,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:15,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:16,318 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:04:16,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:17,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:18,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:19,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:20,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:20,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:04:20,615 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:04:20,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:21,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:22,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:23,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:24,352 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:04:24,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:25,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:25,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:26,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:27,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:28,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:29,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:30,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:31,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:31,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:32,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:33,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:34,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:35,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:35,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:04:35,624 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:04:36,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:37,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:37,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:38,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:04:38,266 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:04:38,267 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:04:38,269 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:04:38,376 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:04:38,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:39,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:40,378 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:04:40,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:41,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:42,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:42,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:43,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:44,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:45,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:46,323 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:04:46,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:47,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:48,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:48,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:49,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:50,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:04:50,622 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:04:50,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:51,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:52,385 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:04:52,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:53,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:53,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:54,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:55,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:56,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:57,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:58,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:04:58,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:04:59,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:00,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:01,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:02,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:03,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:03,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:04,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:05,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:05:05,621 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:05:05,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:05,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:05:05,848 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:05:05,848 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:05:05,850 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:05:06,404 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:05:06,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:07,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:08,405 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:05:08,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:08,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:09,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:10,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:11,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:12,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:13,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:13,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:14,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:15,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:16,327 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:05:16,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:17,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:18,412 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:05:18,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:19,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:19,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:20,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:05:20,622 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:05:20,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:21,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:22,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:23,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:24,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:25,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:25,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:26,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:27,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:28,692 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:29,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:30,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:30,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:31,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:32,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:33,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:05:33,581 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:05:33,581 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:05:33,583 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:05:34,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:34,439 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:05:34,439 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:05:34,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:35,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:05:35,622 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:05:35,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:36,441 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:05:36,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:36,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:37,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:38,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:39,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:40,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:41,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:41,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:42,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:43,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:44,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:45,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:46,331 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:05:46,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:47,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:47,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:48,449 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:05:48,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:49,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:50,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:05:50,622 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:05:50,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:51,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:52,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:52,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:53,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:54,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:55,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:56,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:57,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:57,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:05:58,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:05:59,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:00,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:01,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:02,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:02,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:03,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:04,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:05,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:06:05,606 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:06:05,607 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:06:05,608 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:06:06,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:06,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:06:06,054 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:06:06,470 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:06:06,470 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:06:07,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:08,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:08,471 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:06:08,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:09,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:10,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:11,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:12,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:13,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:14,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:14,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:15,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:16,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:16,335 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:06:17,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:18,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:18,479 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:06:19,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:19,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:20,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:20,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:06:20,751 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:06:21,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:22,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:23,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:24,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:25,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:25,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:26,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:27,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:28,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:29,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:30,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:30,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:31,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:32,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:33,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:34,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:34,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:06:34,484 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:06:34,485 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:06:34,486 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:06:34,501 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:06:35,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:35,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:06:35,751 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:06:36,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:36,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:36,503 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:06:37,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:38,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:39,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:40,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:41,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:41,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:42,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:43,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:44,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:45,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:46,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:46,339 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:06:47,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:47,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:48,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:48,511 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:06:49,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:50,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:50,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:06:50,751 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:06:51,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:52,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:52,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:53,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:54,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:55,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:56,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:57,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:57,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:06:58,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:06:59,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:00,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:01,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:02,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:02,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:03,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:04,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:04,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:07:04,440 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:07:04,440 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:07:04,442 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:07:04,536 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:07:05,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:05,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:07:05,753 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:07:06,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:06,538 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:07:07,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:08,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:08,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:09,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:10,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:11,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:12,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:13,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:14,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:14,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:15,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:16,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:16,343 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:07:17,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:18,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:19,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:20,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:20,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:20,549 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:07:20,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:07:20,752 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:07:21,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:22,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:23,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:24,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:25,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:25,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:26,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:27,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:28,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:29,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:30,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:31,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:31,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:32,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:33,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:34,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:35,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:35,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:07:35,763 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:07:36,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:37,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:37,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:07:37,148 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:07:37,149 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:07:37,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:37,150 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:07:37,573 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:07:38,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:38,574 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:07:39,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:40,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:41,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:42,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:42,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:43,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:44,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:45,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:46,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:46,347 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:07:47,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:48,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:48,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:49,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:50,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:50,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:07:50,759 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:07:51,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:51,582 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:07:52,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:53,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:53,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:54,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:55,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:56,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:57,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:58,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:07:58,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:07:59,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:00,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:01,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:02,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:03,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:03,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:04,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:05,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:05,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:08:05,771 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:08:06,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:07,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:08:07,034 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:08:07,035 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:08:07,036 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:08:07,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:07,598 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:08:08,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:09,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:09,599 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:08:09,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:10,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:11,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:12,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:13,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:14,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:15,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:15,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:16,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:16,351 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:08:17,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:18,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:19,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:20,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:20,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:08:20,764 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:08:21,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:21,606 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:08:21,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:22,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:23,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:24,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:25,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:26,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:26,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:27,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:28,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:29,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:30,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:31,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:31,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:32,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:33,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:34,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:35,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:35,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:08:35,770 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:08:36,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:37,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:37,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:08:37,774 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:08:37,775 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:08:37,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:37,776 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:08:38,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:38,622 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:08:39,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:39,623 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:08:40,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:41,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:41,624 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:08:42,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:43,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:43,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:44,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:45,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:46,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:46,355 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:08:47,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:48,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:49,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:49,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:50,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:50,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:08:50,768 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:08:51,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:52,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:53,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:54,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:08:54,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:55,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:55,633 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:08:56,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:57,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:58,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:08:59,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:00,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:00,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:01,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:02,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:03,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:04,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:05,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:05,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:05,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:09:05,775 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:09:06,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:07,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:08,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:09,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:10,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:10,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:10,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:09:10,618 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:09:10,618 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:09:10,619 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:09:10,652 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:09:11,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:12,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:13,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:13,654 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:09:14,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:15,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:15,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:16,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:16,359 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:09:17,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:18,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:19,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:20,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:20,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:09:20,772 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:09:20,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:21,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:22,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:23,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:24,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:25,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:26,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:26,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:27,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:27,662 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:09:28,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:29,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:30,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:31,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:32,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:32,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:33,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:34,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:35,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:35,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:09:35,772 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:09:36,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:37,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:37,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:38,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:39,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:40,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:41,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:42,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:43,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:43,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:43,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:09:43,366 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:09:43,367 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:09:43,368 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:09:43,682 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:09:44,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:45,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:45,683 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:09:46,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:46,363 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:09:47,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:48,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:49,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:49,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:50,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:50,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:09:50,772 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:09:51,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:52,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:53,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:54,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:55,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:09:55,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:56,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:57,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:57,691 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:09:58,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:09:59,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:00,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:00,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:01,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:02,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:03,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:04,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:05,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:05,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:10:05,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:10:05,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:06,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:07,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:08,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:09,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:10,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:11,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:11,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:12,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:13,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:13,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:10:13,955 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:10:13,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:10:13,956 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:10:14,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:14,711 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:10:15,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:15,712 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:10:16,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:16,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:16,366 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:10:17,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:17,713 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:10:18,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:19,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:20,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:20,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:10:20,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:10:21,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:21,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:22,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:23,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:24,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:25,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:26,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:27,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:27,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:28,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:29,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:29,721 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:10:30,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:31,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:32,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:32,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:33,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:34,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:35,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:35,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:10:35,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:10:36,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:37,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:37,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:38,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:39,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:40,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:41,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:42,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:43,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:43,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:43,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:10:43,907 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:10:43,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:10:43,909 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:10:44,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:44,743 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:10:45,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:45,744 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:10:46,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:46,370 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:10:47,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:48,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:48,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:49,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:50,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:50,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:10:50,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:10:51,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:52,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:53,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:54,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:54,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:55,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:56,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:57,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:57,751 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:10:58,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:10:59,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:10:59,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:00,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:01,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:02,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:03,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:04,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:05,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:05,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:05,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:11:05,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:11:06,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:07,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:08,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:09,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:10,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:10,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:11,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:12,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:13,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:14,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:11:14,180 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:11:14,181 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:11:14,182 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:11:14,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:14,771 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:11:15,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:15,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:15,772 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:11:16,374 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:11:16,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:17,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:17,773 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:11:18,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:19,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:20,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:20,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:11:20,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:11:20,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:21,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:22,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:23,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:24,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:25,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:26,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:26,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:27,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:28,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:29,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:29,784 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:11:30,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:31,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:31,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:32,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:33,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:34,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:35,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:35,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:11:35,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:11:36,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:36,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:37,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:38,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:39,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:40,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:41,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:42,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:42,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:43,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:44,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:11:44,025 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:11:44,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:11:44,027 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:11:44,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:44,801 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:11:45,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:45,802 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:11:46,378 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:11:46,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:47,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:47,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:48,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:49,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:50,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:50,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:11:50,773 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:11:51,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:52,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:52,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:53,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:54,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:55,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:56,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:57,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:58,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:11:58,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:59,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:11:59,813 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:12:00,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:01,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:02,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:03,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:03,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:04,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:05,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:05,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:12:05,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:12:06,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:07,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:08,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:08,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:09,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:10,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:11,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:12,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:13,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:13,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:12:13,785 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:12:13,786 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:12:13,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:13,787 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:12:13,828 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:12:14,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:15,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:15,830 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:12:16,382 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:12:16,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:17,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:18,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:19,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:19,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:20,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:20,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:12:20,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:12:21,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:22,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:23,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:24,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:25,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:25,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:26,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:27,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:28,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:29,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:30,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:30,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:31,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:31,840 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:12:32,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:33,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:34,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:35,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:35,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:12:35,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:12:35,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:36,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:37,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:38,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:39,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:40,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:41,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:41,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:42,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:43,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:44,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:45,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:46,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:46,390 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:12:46,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:47,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:48,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:12:48,120 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:12:48,120 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:12:48,122 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:12:48,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:48,860 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:12:49,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:49,861 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:12:50,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:50,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:12:50,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:12:51,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:51,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:52,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:53,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:54,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:55,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:56,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:57,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:12:57,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:58,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:12:59,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:00,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:01,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:02,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:02,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:03,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:03,871 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:13:04,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:05,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:05,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:13:05,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:13:06,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:07,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:07,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:08,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:09,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:10,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:11,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:12,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:13,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:13,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:14,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:15,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:16,397 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:13:16,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:17,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:18,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:13:18,167 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:13:18,167 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:13:18,168 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:13:18,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:18,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:18,893 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:13:19,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:19,894 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:13:20,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:20,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:13:20,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:13:21,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:22,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:23,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:24,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:24,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:25,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:26,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:27,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:28,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:29,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:30,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:30,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:31,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:32,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:33,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:34,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:35,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:35,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:13:35,774 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:13:35,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:35,904 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:13:36,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:37,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:38,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:39,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:40,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:41,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:41,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:42,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:43,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:44,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:45,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:46,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:46,393 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:13:46,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:47,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:48,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:49,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:13:49,791 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:13:49,791 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:13:49,796 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:13:49,928 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:13:50,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:50,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:13:50,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:13:51,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:51,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:51,930 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:13:52,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:53,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:54,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:55,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:56,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:57,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:57,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:13:58,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:13:59,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:00,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:01,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:02,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:02,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:03,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:03,938 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:14:04,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:05,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:05,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:14:05,776 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:14:06,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:07,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:07,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:08,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:09,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:10,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:11,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:12,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:13,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:13,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:14,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:15,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:16,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:16,402 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:14:17,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:18,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:18,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:19,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:14:19,068 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:14:19,069 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:14:19,071 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:14:19,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:19,961 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:14:20,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:20,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:14:20,776 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:14:21,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:21,963 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:14:22,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:23,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:24,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:24,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:25,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:26,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:27,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:28,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:29,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:30,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:30,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:31,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:32,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:33,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:33,971 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:14:34,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:35,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:35,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:35,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:14:35,776 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:14:36,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:37,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:38,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:39,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:40,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:41,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:41,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:42,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:43,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:44,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:45,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:46,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:46,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:46,406 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:14:47,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:48,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:49,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:50,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:14:50,060 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:14:50,061 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:14:50,062 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:14:50,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:50,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:14:50,776 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:14:50,989 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:14:51,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:51,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:51,990 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:14:52,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:53,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:54,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:55,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:56,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:57,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:14:57,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:58,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:14:59,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:00,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:01,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:02,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:02,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:03,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:03,997 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:15:04,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:05,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:05,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:15:05,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:15:06,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:07,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:07,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:08,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:09,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:10,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:11,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:12,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:13,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:13,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:14,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:15,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:16,408 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:15:16,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:17,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:18,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:18,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:19,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:15:19,357 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:15:19,357 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:15:19,359 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:15:19,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:20,018 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:15:20,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:20,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:15:20,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:15:21,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:22,019 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:15:22,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:23,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:23,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:24,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:25,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:26,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:27,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:28,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:29,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:29,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:30,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:31,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:32,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:33,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:34,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:34,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:35,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:15:35,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:15:35,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:36,029 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:15:36,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:37,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:38,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:39,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:40,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:40,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:41,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:42,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:43,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:44,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:45,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:46,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:46,416 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:15:46,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:47,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:48,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:49,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:50,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:15:50,778 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:15:50,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:51,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:15:51,286 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:15:51,287 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:15:51,290 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:15:51,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:51,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:52,044 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:15:52,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:53,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:54,046 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:15:54,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:55,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:56,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:57,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:15:57,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:58,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:15:59,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:00,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:01,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:02,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:03,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:03,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:04,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:05,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:16:05,778 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:16:05,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:06,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:07,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:08,055 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:16:08,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:08,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:09,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:10,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:11,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:12,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:13,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:14,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:14,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:15,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:16,410 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:16:16,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:17,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:18,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:19,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:19,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:20,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:16:20,784 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:16:20,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:21,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:16:21,195 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:16:21,196 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:16:21,197 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:16:21,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:22,072 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:16:22,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:23,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:24,074 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:16:24,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:25,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:25,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:26,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:27,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:28,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:29,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:30,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:31,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:31,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:32,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:33,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:34,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:35,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:16:35,784 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:16:35,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:36,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:36,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:37,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:38,084 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:16:38,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:39,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:40,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:41,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:42,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:42,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:43,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:44,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:45,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:46,415 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:16:46,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:47,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:47,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:48,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:49,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:50,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:16:50,785 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:16:50,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:51,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:52,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:52,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:53,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:16:53,286 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:16:53,287 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:16:53,292 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:16:53,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:54,104 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:16:54,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:55,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:56,106 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:16:56,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:57,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:58,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:16:58,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:16:59,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:00,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:01,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:02,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:03,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:04,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:04,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:05,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:17:05,786 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:17:05,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:06,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:07,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:08,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:09,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:09,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:10,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:11,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:12,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:13,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:14,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:14,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:15,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:16,419 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:17:17,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:19,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:20,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:20,125 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:17:25,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:27,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:27,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:17:27,865 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:17:30,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:35,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:35,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:37,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:38,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:41,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:46,424 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:17:46,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:17:46,586 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:17:46,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:46,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:51,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:17:54,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:55,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:17:57,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:02,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:04,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:18:04,337 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:18:04,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:06,118 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:07,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:12,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:16,428 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:18:17,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:19,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:19,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:18:19,987 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:18:20,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:20,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:21,498 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:18:21,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:22,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:23,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:23,500 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:18:23,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:24,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:26,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:27,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:28,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:28,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:29,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:30,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:31,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:32,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:33,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:33,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:34,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:34,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:18:34,985 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:18:35,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:36,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:37,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:38,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:39,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:39,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:40,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:41,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:42,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:43,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:44,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:44,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:45,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:46,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:46,431 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:18:47,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:48,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:49,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:49,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:18:49,985 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:18:50,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:50,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:51,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:52,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:53,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:54,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:55,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:56,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:18:56,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:57,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:58,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:18:59,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:00,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:01,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:01,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:02,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:03,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:04,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:04,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:19:04,985 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:19:05,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:06,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:06,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:07,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:08,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:09,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:10,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:12,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:16,436 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:19:17,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:22,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:27,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:32,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:32,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:19:32,083 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:19:33,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:33,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:34,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:35,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:36,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:37,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:38,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:39,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:39,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:40,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:41,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:42,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:43,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:44,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:45,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:45,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:46,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:46,440 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:19:47,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:19:47,075 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:19:47,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:48,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:49,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:50,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:50,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:51,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:52,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:53,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:54,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:55,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:56,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:19:56,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:57,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:58,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:19:59,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:00,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:01,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:02,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:20:02,076 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:20:02,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:02,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:03,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:04,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:05,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:06,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:07,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:08,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:08,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:09,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:10,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:11,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:12,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:13,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:14,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:14,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:15,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:16,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:16,444 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:20:17,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:20:17,076 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:20:17,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:18,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:19,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:19,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:20,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:21,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:22,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:23,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:24,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:25,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:25,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:26,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:27,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:28,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:29,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:30,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:30,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:31,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:32,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:20:32,076 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:20:32,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:33,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:34,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:35,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:35,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:36,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:37,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:38,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:39,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:40,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:41,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:41,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:42,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:43,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:44,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:45,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:46,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:46,449 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:20:46,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:47,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:20:47,076 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:20:47,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:48,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:49,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:50,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:51,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:52,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:52,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:53,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:54,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:55,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:56,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:57,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:20:57,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:58,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:20:59,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:00,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:01,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:02,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:21:02,076 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:21:02,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:02,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:03,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:04,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:05,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:06,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:07,089 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:08,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:08,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:09,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:10,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:11,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:12,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:13,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:14,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:14,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:15,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:16,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:16,453 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:21:19,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:24,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:29,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:34,458 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:39,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:44,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:46,454 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:21:50,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:55,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:21:57,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:57,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:21:57,636 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:21:58,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:21:59,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:00,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:01,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:01,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:02,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:02,663 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:22:03,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:04,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:04,665 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:22:05,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:06,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:07,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:07,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:08,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:09,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:10,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:11,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:12,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:12,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:22:12,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:22:12,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:13,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:14,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:14,671 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:22:15,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:16,458 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:22:16,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:17,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:17,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:18,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:19,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:20,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:21,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:22,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:22,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:23,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:24,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:25,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:26,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:27,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:27,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:22:27,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:27,640 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:22:28,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:29,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:30,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:22:30,446 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:22:30,447 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:22:30,448 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:22:30,700 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:22:30,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:31,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:32,702 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:22:32,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:33,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:33,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:34,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:35,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:36,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:37,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:38,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:39,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:39,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:40,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:41,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:42,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:22:42,638 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:22:42,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:43,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:44,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:44,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:45,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:46,462 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:22:46,711 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:22:46,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:47,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:48,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:49,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:50,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:50,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:51,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:52,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:53,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:54,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:55,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:56,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:22:56,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:57,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:22:57,648 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:22:57,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:58,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:22:59,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:00,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:01,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:01,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:02,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:23:02,750 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:23:02,751 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:23:02,752 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:23:03,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:03,735 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:23:03,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:04,737 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:23:04,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:05,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:06,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:07,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:07,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:08,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:09,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:10,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:11,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:12,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:12,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:23:12,642 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:23:12,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:13,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:14,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:15,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:16,466 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:23:16,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:17,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:17,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:18,745 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:23:18,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:19,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:20,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:21,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:22,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:22,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:23,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:24,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:25,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:26,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:27,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:27,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:23:27,659 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:23:27,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:28,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:29,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:30,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:31,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:32,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:33,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:23:33,548 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:23:33,549 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:23:33,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:33,551 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:23:33,763 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:23:33,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:34,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:35,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:36,765 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:23:36,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:37,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:38,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:38,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:39,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:40,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:41,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:42,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:23:42,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:23:42,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:43,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:43,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:44,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:45,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:46,470 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:23:46,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:47,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:48,772 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:23:48,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:49,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:49,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:50,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:51,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:52,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:53,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:54,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:55,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:23:55,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:56,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:57,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:23:57,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:23:57,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:58,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:23:59,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:00,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:00,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:01,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:02,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:03,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:04,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:05,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:24:05,381 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:24:05,381 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:24:05,383 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:24:05,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:05,797 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:24:05,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:06,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:07,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:08,799 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:24:08,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:09,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:10,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:11,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:11,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:12,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:24:12,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:24:12,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:13,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:14,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:15,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:16,474 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:24:16,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:17,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:17,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:18,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:19,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:20,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:21,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:22,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:22,808 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:24:22,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:23,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:24,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:25,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:26,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:27,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:27,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:24:27,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:24:27,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:28,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:29,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:30,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:32,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:32,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:33,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:34,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:35,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:36,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:37,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:38,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:38,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:24:38,310 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:24:38,311 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:24:38,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:38,313 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:24:38,824 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:24:39,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:40,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:40,825 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:24:41,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:42,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:42,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:24:42,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:24:43,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:43,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:44,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:45,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:46,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:46,479 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:24:47,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:48,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:49,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:49,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:50,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:51,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:52,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:53,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:54,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:54,834 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:24:55,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:55,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:24:56,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:57,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:57,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:24:57,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:24:58,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:24:59,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:00,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:00,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:01,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:02,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:03,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:04,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:05,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:06,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:06,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:07,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:08,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:09,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:10,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:11,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:11,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:11,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:25:11,762 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:25:11,762 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:25:11,764 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:25:11,849 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:25:12,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:12,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:25:12,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:25:13,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:14,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:14,851 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:25:15,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:16,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:16,482 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:25:17,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:17,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:18,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:19,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:20,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:21,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:22,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:22,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:23,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:24,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:25,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:26,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:26,860 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:25:27,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:27,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:25:27,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:25:27,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:28,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:29,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:30,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:31,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:32,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:33,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:33,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:34,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:35,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:36,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:37,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:38,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:38,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:39,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:40,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:41,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:25:41,071 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:25:41,071 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:25:41,073 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:25:41,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:41,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:25:42,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:42,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:25:42,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:25:42,882 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:25:43,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:43,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:44,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:45,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:46,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:46,486 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:25:47,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:48,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:49,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:49,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:50,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:51,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:52,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:53,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:54,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:55,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:55,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:25:56,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:56,891 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:25:57,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:57,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:25:57,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:25:58,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:25:59,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:00,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:00,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:01,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:02,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:03,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:04,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:05,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:06,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:06,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:07,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:08,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:09,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:10,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:10,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:26:10,945 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:26:10,945 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:26:10,947 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:26:11,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:11,913 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:26:12,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:12,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:12,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:26:12,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:26:12,914 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:26:13,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:14,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:15,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:16,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:16,490 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:26:17,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:17,492 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:18,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:19,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:20,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:21,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:22,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:22,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:23,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:24,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:25,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:26,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:26,923 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:26:27,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:27,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:26:27,647 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:26:27,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:28,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:29,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:30,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:31,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:32,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:33,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:33,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:34,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:35,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:36,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:37,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:38,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:38,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:39,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:40,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:41,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:41,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:26:41,613 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:26:41,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:26:41,615 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:26:41,944 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:26:42,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:42,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:26:42,647 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:26:43,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:43,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:44,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:44,946 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:26:45,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:46,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:46,493 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:26:47,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:48,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:49,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:49,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:50,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:51,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:52,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:53,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:54,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:55,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:55,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:26:56,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:57,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:57,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:26:57,647 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:26:58,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:26:58,955 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:26:59,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:00,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:00,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:01,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:02,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:03,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:04,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:05,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:06,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:06,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:07,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:08,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:09,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:10,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:11,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:11,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:12,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:12,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:27:12,650 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:27:13,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:14,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:15,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:27:15,213 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:27:15,214 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:27:15,216 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:27:15,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:15,975 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:27:16,498 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:27:16,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:16,976 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:27:17,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:17,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:18,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:19,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:20,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:21,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:22,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:22,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:23,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:24,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:25,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:26,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:27,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:27,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:27:27,650 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:27:27,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:28,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:28,983 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:27:29,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:30,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:31,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:32,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:33,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:33,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:34,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:35,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:36,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:37,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:38,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:38,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:39,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:40,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:41,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:42,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:42,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:27:42,653 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:27:43,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:43,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:44,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:27:44,600 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:27:44,600 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:27:44,602 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:27:45,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:45,008 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:27:45,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:46,502 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:27:46,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:47,009 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:27:47,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:48,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:49,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:49,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:50,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:51,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:52,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:53,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:54,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:27:54,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:55,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:56,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:57,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:27:57,653 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:27:57,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:58,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:59,017 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:27:59,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:27:59,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:00,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:01,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:02,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:03,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:04,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:05,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:05,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:06,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:07,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:08,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:09,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:10,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:10,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:11,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:12,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:28:12,660 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:28:12,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:13,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:14,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:15,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:15,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:15,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:28:15,803 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:28:15,804 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:28:15,805 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:28:16,040 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:28:16,506 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:28:16,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:17,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:18,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:19,043 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:28:19,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:20,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:21,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:21,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:22,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:23,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:24,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:25,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:26,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:27,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:28:27,657 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:28:27,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:27,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:28,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:29,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:30,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:31,050 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:28:31,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:32,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:33,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:33,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:34,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:35,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:36,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:37,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:38,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:39,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:39,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:40,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:41,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:42,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:28:42,659 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:28:42,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:43,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:44,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:44,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:45,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:28:45,593 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:28:45,594 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:28:45,596 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:28:46,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:46,066 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:28:46,510 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:28:47,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:47,067 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:28:48,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:49,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:49,068 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:28:50,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:50,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:51,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:52,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:53,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:54,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:55,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:56,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:56,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:28:57,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:57,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:28:57,659 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:28:58,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:59,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:28:59,073 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:29:00,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:01,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:01,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:02,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:03,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:04,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:05,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:06,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:07,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:07,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:08,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:09,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:10,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:11,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:12,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:12,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:29:12,667 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:29:12,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:13,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:14,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:14,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:29:14,257 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:29:14,258 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:29:14,259 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:29:15,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:15,091 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:29:16,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:16,514 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:29:17,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:17,093 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:29:18,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:18,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:19,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:20,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:21,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:22,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:23,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:24,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:24,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:25,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:26,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:27,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:27,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:29:27,664 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:29:28,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:29,101 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:29:29,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:29,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:30,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:31,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:32,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:33,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:34,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:35,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:35,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:36,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:37,117 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:38,118 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:39,118 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:40,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:40,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:41,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:42,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:42,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:29:42,670 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:29:43,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:44,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:45,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:45,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:29:45,249 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:29:45,250 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:29:45,252 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:29:46,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:46,122 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:29:46,518 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:29:46,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:47,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:47,123 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:29:48,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:49,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:50,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:51,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:51,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:52,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:53,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:54,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:55,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:56,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:57,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:57,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:29:57,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:29:57,670 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:29:58,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:29:59,132 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:29:59,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:00,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:01,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:02,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:02,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:03,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:04,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:05,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:06,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:07,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:08,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:08,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:09,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:10,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:11,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:12,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:12,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:30:12,394 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:30:12,394 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:30:12,396 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:30:12,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:30:12,802 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:30:13,149 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:30:13,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:13,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:14,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:15,150 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:30:15,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:16,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:16,522 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:30:17,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:18,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:19,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:19,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:20,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:21,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:22,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:23,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:24,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:25,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:25,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:26,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:27,157 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:30:27,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:27,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:30:27,798 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:30:28,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:29,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:30,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:30,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:31,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:32,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:33,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:34,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:35,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:36,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:36,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:37,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:38,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:39,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:40,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:41,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:30:41,213 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:30:41,214 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:30:41,215 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:30:41,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:42,176 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:30:42,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:42,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:42,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:30:42,798 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:30:43,177 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:30:43,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:44,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:45,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:46,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:46,526 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:30:47,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:47,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:48,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:49,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:50,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:51,355 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:52,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:53,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:53,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:54,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:55,184 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:30:55,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:56,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:57,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:57,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:30:57,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:30:58,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:30:58,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:30:59,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:00,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:01,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:02,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:03,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:04,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:04,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:05,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:06,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:07,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:08,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:09,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:09,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:10,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:10,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:31:10,941 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:31:10,941 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:31:10,943 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:31:11,211 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:31:11,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:12,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:12,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:31:12,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:31:13,213 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:31:13,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:14,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:14,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:15,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:16,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:16,530 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:31:17,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:18,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:19,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:20,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:20,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:21,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:22,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:23,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:24,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:25,222 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:31:25,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:26,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:26,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:27,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:27,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:31:27,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:31:28,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:29,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:30,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:31,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:31,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:32,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:33,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:34,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:35,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:36,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:37,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:37,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:38,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:38,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:31:38,604 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:31:38,604 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:31:38,605 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:31:39,239 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:31:39,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:40,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:41,241 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:31:41,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:42,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:42,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:31:42,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:31:42,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:43,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:44,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:45,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:46,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:46,534 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:31:47,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:48,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:48,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:49,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:50,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:51,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:52,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:53,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:53,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:54,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:55,251 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:31:55,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:56,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:57,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:57,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:31:57,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:31:58,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:31:58,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:31:59,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:00,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:01,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:02,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:03,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:04,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:04,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:05,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:06,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:07,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:08,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:09,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:09,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:10,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:11,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:11,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:32:11,448 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:32:11,448 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:32:11,449 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:32:12,271 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:32:12,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:12,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:32:12,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:32:13,272 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:32:13,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:14,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:14,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:15,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:16,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:16,537 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:32:17,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:18,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:19,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:20,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:20,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:21,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:22,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:23,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:24,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:25,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:25,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:26,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:27,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:27,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:32:27,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:32:28,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:29,283 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:32:29,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:30,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:30,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:31,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:32,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:33,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:34,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:35,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:36,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:36,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:37,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:38,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:39,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:40,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:41,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:41,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:42,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:42,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:32:42,803 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:32:43,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:44,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:32:44,054 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:32:44,054 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:32:44,056 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:32:44,301 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:32:44,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:45,303 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:32:45,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:46,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:46,541 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:32:47,304 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:32:47,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:47,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:48,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:49,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:50,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:51,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:52,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:52,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:53,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:54,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:55,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:56,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:57,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:57,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:32:57,803 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:32:57,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:32:58,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:32:59,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:00,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:01,313 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:33:01,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:02,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:03,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:03,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:04,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:05,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:06,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:07,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:08,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:09,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:09,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:10,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:11,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:12,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:12,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:33:12,806 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:33:13,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:14,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:14,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:15,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:16,545 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:33:16,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:17,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:33:17,665 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:33:17,666 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:33:17,668 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:33:18,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:18,337 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:33:19,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:19,338 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:33:20,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:20,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:21,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:22,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:23,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:24,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:25,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:26,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:26,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:27,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:27,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:33:27,806 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:33:28,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:29,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:30,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:31,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:31,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:32,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:33,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:33,348 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:33:34,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:35,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:36,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:37,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:37,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:38,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:39,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:40,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:41,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:42,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:42,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:33:42,806 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:33:42,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:43,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:44,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:45,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:46,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:46,553 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:33:47,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:48,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:48,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:49,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:50,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:50,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:33:50,653 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:33:50,654 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:33:50,655 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:33:51,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:51,370 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:33:52,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:53,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:53,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:53,371 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:33:54,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:55,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:56,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:57,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:57,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:33:57,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:33:58,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:33:59,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:33:59,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:00,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:01,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:02,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:03,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:04,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:04,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:05,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:05,379 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:34:06,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:07,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:08,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:09,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:10,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:10,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:11,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:12,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:12,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:34:12,806 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:34:13,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:14,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:15,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:16,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:16,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:16,554 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:34:17,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:18,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:19,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:20,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:21,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:21,186 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:22,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:22,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:34:22,791 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:34:22,792 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:34:22,793 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:34:23,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:23,400 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:34:24,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:25,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:25,401 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:34:26,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:26,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:27,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:27,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:34:27,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:34:28,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:29,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:30,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:31,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:32,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:32,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:33,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:34,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:35,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:36,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:37,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:38,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:38,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:39,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:39,410 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:34:40,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:41,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:42,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:42,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:34:42,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:34:43,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:43,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:44,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:45,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:46,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:46,556 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:34:47,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:48,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:48,250 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:49,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:50,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:51,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:52,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:53,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:53,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:54,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:55,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:55,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:34:55,960 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:34:55,960 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:34:55,961 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:34:56,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:56,431 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:34:57,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:57,432 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:34:57,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:34:57,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:34:58,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:59,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:34:59,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:34:59,433 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:35:00,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:01,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:02,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:03,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:04,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:05,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:05,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:06,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:07,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:08,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:09,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:09,439 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:35:10,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:10,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:11,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:12,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:12,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:35:12,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:35:13,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:14,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:15,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:16,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:16,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:16,560 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:35:17,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:18,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:19,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:20,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:21,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:21,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:22,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:23,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:24,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:25,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:25,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:35:25,890 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:35:25,891 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:35:25,892 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:35:26,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:26,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:26,458 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:35:27,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:27,459 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:35:27,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:35:27,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:35:28,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:29,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:30,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:31,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:32,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:32,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:33,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:34,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:35,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:36,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:37,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:37,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:38,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:39,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:40,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:41,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:41,468 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:35:42,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:42,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:35:42,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:35:42,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:43,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:44,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:45,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:46,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:46,565 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:35:47,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:48,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:48,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:49,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:50,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:51,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:52,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:53,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:53,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:54,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:55,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:56,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:57,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:35:57,406 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:35:57,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:35:57,408 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:35:57,486 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:35:57,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:35:57,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:57,864 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:35:58,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:35:59,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:35:59,487 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:35:59,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:00,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:01,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:02,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:03,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:04,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:04,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:05,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:06,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:07,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:08,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:09,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:09,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:10,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:11,495 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:36:11,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:12,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:36:12,858 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:36:12,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:13,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:14,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:15,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:15,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:16,569 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:36:16,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:17,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:18,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:19,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:20,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:21,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:21,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:22,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:23,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:24,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:25,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:26,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:26,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:27,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:36:27,495 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:36:27,496 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:36:27,497 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:36:27,518 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:36:27,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:36:27,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:36:27,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:28,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:29,519 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:36:29,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:30,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:31,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:32,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:32,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:33,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:34,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:35,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:36,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:37,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:38,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:38,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:39,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:40,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:41,527 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:36:41,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:42,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:36:42,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:36:42,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:43,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:43,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:44,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:45,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:46,574 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:36:46,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:47,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:48,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:48,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:49,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:50,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:51,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:52,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:53,270 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:53,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:54,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:55,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:36:55,900 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:36:55,900 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:36:55,902 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:36:56,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:56,546 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:36:57,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:57,547 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:36:57,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:36:57,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:36:58,270 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:36:58,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:36:59,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:00,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:01,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:02,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:03,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:03,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:04,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:05,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:06,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:07,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:08,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:08,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:09,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:10,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:11,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:11,555 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:37:12,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:12,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:37:12,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:37:13,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:14,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:14,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:15,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:16,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:16,577 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:37:17,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:18,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:19,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:19,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:20,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:21,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:22,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:23,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:24,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:24,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:25,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:26,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:26,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:37:26,585 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:37:26,586 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:37:26,587 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:37:27,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:27,577 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:37:27,578 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:37:27,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:37:27,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:37:28,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:29,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:29,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:29,579 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:37:30,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:31,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:32,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:33,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:34,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:34,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:35,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:36,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:37,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:38,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:39,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:39,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:39,585 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:37:40,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:41,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:42,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:42,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:37:42,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:37:43,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:44,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:45,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:45,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:46,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:46,581 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:37:47,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:48,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:49,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:50,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:50,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:51,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:52,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:53,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:54,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:55,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:37:55,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:55,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:37:55,719 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:37:55,720 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:37:55,722 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:37:56,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:56,609 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:37:57,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:57,610 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:37:57,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:37:57,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:37:58,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:37:59,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:00,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:01,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:01,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:02,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:03,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:04,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:05,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:06,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:06,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:07,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:08,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:09,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:09,617 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:38:10,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:11,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:11,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:12,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:12,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:38:12,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:38:13,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:14,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:15,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:16,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:16,585 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:38:16,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:17,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:18,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:19,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:20,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:21,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:22,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:22,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:23,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:24,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:24,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:38:24,716 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:38:24,716 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:38:24,718 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:38:25,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:25,636 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:38:26,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:27,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:27,638 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:38:27,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:38:27,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:38:28,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:28,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:29,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:30,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:31,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:32,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:33,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:33,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:34,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:35,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:36,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:37,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:37,644 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:38:38,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:38,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:39,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:40,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:41,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:42,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:42,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:38:42,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:38:43,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:44,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:44,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:45,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:46,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:46,589 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:38:47,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:48,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:49,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:49,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:50,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:51,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:52,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:53,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:54,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:38:54,114 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:38:54,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:38:54,116 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:38:54,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:38:54,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:54,663 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:38:55,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:55,664 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:38:56,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:57,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:57,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:38:57,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:38:58,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:38:59,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:00,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:00,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:01,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:02,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:03,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:04,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:05,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:06,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:06,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:07,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:07,672 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:39:08,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:09,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:10,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:11,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:11,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:12,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:12,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:39:12,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:39:13,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:14,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:15,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:16,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:16,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:16,594 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:39:17,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:18,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:19,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:20,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:21,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:21,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:22,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:23,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:24,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:39:24,384 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:39:24,384 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:39:24,386 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:39:24,694 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:39:24,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:25,695 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:39:25,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:26,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:26,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:27,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:27,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:39:27,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:39:28,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:29,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:30,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:31,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:32,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:32,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:33,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:34,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:35,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:37,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:38,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:38,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:39,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:39,704 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:39:40,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:41,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:42,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:42,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:39:42,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:39:43,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:43,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:44,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:45,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:46,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:46,597 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:39:47,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:48,006 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:48,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:49,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:50,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:51,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:52,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:53,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:53,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:54,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:54,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:39:54,718 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:39:54,719 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:39:54,721 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:39:54,723 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:39:55,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:55,724 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:39:56,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:57,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:57,726 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:39:57,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:39:57,910 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:39:58,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:39:58,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:39:59,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:00,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:01,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:02,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:03,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:04,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:04,287 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:05,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:06,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:07,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:07,733 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:40:08,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:09,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:09,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:10,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:11,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:12,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:12,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:40:12,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:40:13,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:14,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:15,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:15,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:16,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:16,601 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:40:17,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:18,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:19,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:20,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:20,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:21,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:22,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:23,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:23,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:40:23,995 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:40:23,996 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:40:23,997 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:40:24,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:24,753 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:40:25,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:25,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:25,754 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:40:26,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:27,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:27,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:40:27,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:40:28,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:29,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:30,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:31,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:31,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:32,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:33,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:34,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:35,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:36,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:37,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:37,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:38,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:39,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:39,763 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:40:40,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:41,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:42,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:42,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:42,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:40:42,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:40:43,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:44,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:45,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:46,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:46,605 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:40:47,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:47,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:48,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:49,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:50,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:51,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:52,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:53,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:53,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:54,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:55,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:55,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:40:55,736 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:40:55,736 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:40:55,737 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:40:55,785 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:40:56,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:57,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:57,787 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:40:57,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:40:57,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:40:58,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:40:58,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:40:59,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:00,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:01,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:02,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:03,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:04,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:04,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:05,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:06,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:07,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:08,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:09,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:09,796 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:41:10,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:10,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:11,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:12,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:12,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:41:12,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:41:13,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:14,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:15,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:15,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:16,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:16,609 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:41:17,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:18,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:19,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:20,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:20,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:21,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:22,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:23,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:24,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:25,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:25,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:25,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:41:25,535 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:41:25,535 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:41:25,537 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:41:25,814 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:41:26,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:27,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:27,816 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:41:27,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:41:27,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:41:28,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:29,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:30,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:31,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:31,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:32,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:33,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:34,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:35,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:36,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:37,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:37,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:38,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:39,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:39,823 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:41:40,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:41,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:42,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:42,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:41:42,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:41:42,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:43,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:44,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:45,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:46,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:46,613 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:41:47,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:48,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:48,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:49,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:50,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:51,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:52,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:53,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:53,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:54,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:55,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:56,105 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:41:56,107 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:41:56,107 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:41:56,112 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:41:56,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:56,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:41:57,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:57,843 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:41:57,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:41:57,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:41:58,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:41:58,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:41:59,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:00,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:01,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:02,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:03,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:04,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:04,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:05,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:06,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:07,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:08,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:09,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:09,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:10,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:11,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:12,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:12,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:42:12,911 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:42:13,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:13,854 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:42:14,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:14,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:15,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:16,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:16,617 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:42:17,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:18,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:19,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:20,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:20,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:21,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:22,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:23,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:24,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:25,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:25,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:26,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:27,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:27,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:42:27,917 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:42:28,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:28,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:42:28,692 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:42:28,693 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:42:28,695 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:42:28,876 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:42:29,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:29,878 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:42:30,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:30,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:31,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:31,879 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:42:32,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:33,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:34,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:35,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:36,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:36,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:37,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:38,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:39,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:40,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:41,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:42,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:42,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:42,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:42:42,917 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:42:43,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:44,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:45,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:45,889 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:42:46,621 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:42:46,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:47,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:47,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:48,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:49,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:50,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:51,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:52,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:53,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:53,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:54,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:55,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:56,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:57,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:57,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:42:57,924 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:42:58,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:42:58,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:42:59,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:00,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:01,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:02,300 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:43:02,301 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:43:02,302 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:43:02,303 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:43:02,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:02,905 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:43:03,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:03,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:43:04,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:04,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:05,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:06,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:07,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:08,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:09,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:10,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:10,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:11,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:12,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:12,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:43:12,921 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:43:13,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:14,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:15,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:15,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:15,914 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:43:16,623 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:43:16,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:17,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:18,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:19,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:20,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:21,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:21,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:22,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:23,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:24,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:25,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:26,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:26,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:27,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:27,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:43:27,921 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:43:28,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:29,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:30,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:31,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:43:31,184 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:43:31,184 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:43:31,186 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:43:31,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:31,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:31,936 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:43:32,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:33,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:33,938 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:43:34,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:35,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:36,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:37,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:37,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:38,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:39,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:40,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:41,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:42,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:42,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:43:42,921 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:43:42,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:43,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:44,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:45,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:45,946 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:43:46,627 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:43:46,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:47,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:48,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:48,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:49,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:50,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:51,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:52,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:53,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:53,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:54,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:55,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:56,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:57,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:43:57,933 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:43:57,969 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:58,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:43:58,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:43:59,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:00,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:01,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:44:01,331 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:44:01,332 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:44:01,333 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:44:01,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:01,963 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:44:02,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:03,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:03,965 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:44:04,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:04,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:05,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:06,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:07,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:08,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:09,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:10,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:10,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:11,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:12,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:44:12,928 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:44:12,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:13,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:14,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:15,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:15,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:15,972 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:44:16,632 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:44:16,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:17,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:18,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:19,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:20,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:21,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:21,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:22,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:23,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:24,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:25,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:26,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:27,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:27,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:44:27,948 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:44:27,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:28,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:29,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:30,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:31,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:32,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:32,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:33,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:44:33,730 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:44:33,731 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:44:33,732 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:44:33,999 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:44:34,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:35,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:36,001 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:44:36,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:37,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:38,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:38,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:39,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:40,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:41,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:42,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:42,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:44:42,933 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:44:43,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:43,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:44,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:45,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:46,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:46,635 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:44:47,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:48,008 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:44:48,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:48,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:49,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:50,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:51,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:52,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:53,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:54,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:54,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:44:55,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:56,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:57,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:57,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:44:57,942 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:44:58,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:59,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:44:59,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:00,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:01,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:02,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:03,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:03,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:45:03,250 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:45:03,250 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:45:03,252 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:45:04,034 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:45:04,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:04,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:05,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:06,035 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:45:06,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:07,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:08,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:09,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:10,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:10,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:11,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:12,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:12,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:45:12,937 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:45:13,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:14,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:15,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:15,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:16,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:16,639 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:45:17,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:18,043 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:45:18,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:19,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:20,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:21,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:21,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:22,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:23,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:24,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:25,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:26,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:27,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:27,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:27,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:45:27,949 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:45:28,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:29,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:30,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:31,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:32,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:32,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:33,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:33,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:45:33,309 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:45:33,309 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:45:33,311 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:45:34,061 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:45:34,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:35,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:36,062 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:45:36,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:37,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:37,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:38,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:39,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:40,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:41,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:42,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:42,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:45:42,944 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:45:43,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:43,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:44,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:45,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:46,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:46,643 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:45:47,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:48,070 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:45:48,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:48,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:49,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:50,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:51,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:52,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:53,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:53,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:54,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:55,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:56,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:57,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:57,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:45:57,966 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:45:58,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:45:58,458 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:45:59,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:00,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:01,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:02,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:03,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:03,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:46:03,798 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:46:03,799 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:46:03,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:03,801 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:46:04,099 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:46:04,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:05,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:06,101 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:46:06,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:07,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:08,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:09,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:09,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:10,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:11,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:12,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:12,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:46:12,950 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:46:13,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:14,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:15,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:15,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:16,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:16,647 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:46:17,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:18,107 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:46:18,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:19,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:20,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:20,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:21,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:22,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:23,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:24,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:25,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:26,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:26,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:27,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:27,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:46:27,950 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:46:28,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:29,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:30,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:31,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:31,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:32,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:33,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:34,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:34,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:46:34,819 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:46:34,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:46:34,821 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:46:35,134 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:46:35,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:36,135 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:46:36,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:37,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:37,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:38,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:39,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:40,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:41,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:42,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:42,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:42,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:46:42,950 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:46:43,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:44,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:45,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:46,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:46,652 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:46:47,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:47,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:48,145 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:46:48,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:49,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:50,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:51,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:52,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:53,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:53,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:54,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:55,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:56,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:57,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:57,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:46:57,962 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:46:58,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:46:58,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:46:59,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:00,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:01,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:02,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:03,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:03,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:47:03,952 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:47:03,952 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:47:03,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:03,954 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:47:04,168 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:47:04,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:05,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:06,170 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:47:06,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:07,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:08,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:09,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:09,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:10,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:11,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:12,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:12,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:47:12,954 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:47:13,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:14,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:15,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:15,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:16,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:16,655 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:47:17,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:18,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:19,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:20,180 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:47:20,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:20,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:21,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:22,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:23,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:24,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:25,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:26,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:26,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:27,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:27,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:47:27,954 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:47:28,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:29,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:30,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:31,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:31,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:32,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:33,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:34,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:35,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:36,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:36,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:37,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:38,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:47:38,190 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:47:38,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:47:38,192 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:47:38,201 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:47:38,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:39,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:40,203 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:47:40,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:41,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:41,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:42,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:42,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:47:42,954 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:47:43,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:44,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:45,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:46,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:46,659 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:47:46,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:47,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:48,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:49,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:50,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:51,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:52,211 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:47:52,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:52,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:53,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:54,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:55,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:56,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:57,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:47:57,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:57,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:47:57,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:47:58,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:47:59,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:00,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:01,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:02,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:03,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:03,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:04,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:05,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:06,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:07,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:08,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:08,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:09,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:10,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:48:10,032 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:48:10,032 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:48:10,033 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:48:10,236 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:48:10,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:11,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:12,238 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:48:12,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:12,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:48:12,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:48:13,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:14,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:14,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:15,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:16,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:16,663 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:48:17,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:18,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:19,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:20,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:20,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:21,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:22,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:23,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:24,246 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:48:24,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:25,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:25,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:26,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:27,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:27,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:48:27,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:48:28,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:29,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:30,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:31,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:31,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:32,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:33,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:34,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:35,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:36,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:36,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:37,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:38,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:39,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:48:39,094 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:48:39,094 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:48:39,095 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:48:39,273 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:48:39,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:40,274 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:48:40,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:41,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:42,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:42,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:42,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:48:42,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:48:43,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:44,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:45,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:46,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:46,667 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:48:47,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:47,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:48,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:49,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:50,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:51,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:52,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:53,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:53,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:54,283 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:48:54,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:55,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:56,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:57,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:57,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:48:57,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:48:58,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:48:58,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:48:59,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:00,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:01,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:02,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:03,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:03,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:04,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:05,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:06,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:07,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:08,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:09,301 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:49:09,303 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:49:09,304 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:49:09,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:09,306 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:49:09,306 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:49:09,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:10,307 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:49:10,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:11,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:12,308 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:49:12,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:12,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:49:12,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:49:13,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:14,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:14,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:15,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:16,671 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:49:16,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:17,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:18,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:19,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:19,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:20,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:21,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:22,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:23,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:24,316 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:49:24,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:24,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:25,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:26,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:27,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:27,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:49:27,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:49:28,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:29,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:30,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:30,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:31,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:32,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:33,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:34,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:35,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:35,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:36,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:37,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:38,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:39,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:40,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:40,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:40,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:49:40,801 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:49:40,802 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:49:40,804 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:49:41,335 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:49:41,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:42,335 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:49:42,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:42,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:49:42,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:49:43,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:44,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:45,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:46,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:46,678 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:49:46,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:47,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:48,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:49,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:50,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:51,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:52,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:52,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:53,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:54,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:55,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:56,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:57,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:57,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:49:57,956 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:49:58,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:49:58,344 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:49:58,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:49:59,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:00,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:01,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:02,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:03,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:03,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:04,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:05,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:06,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:07,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:08,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:08,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:09,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:10,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:11,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:12,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:12,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:50:12,967 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:50:13,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:13,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:14,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:15,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:50:15,714 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:50:15,715 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:50:15,716 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:50:16,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:16,365 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:50:16,679 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:50:17,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:18,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:18,367 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:50:19,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:19,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:20,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:21,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:22,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:23,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:24,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:25,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:25,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:26,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:27,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:27,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:50:27,958 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:50:28,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:29,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:30,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:30,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:31,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:32,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:32,376 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:50:33,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:34,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:35,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:36,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:36,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:37,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:38,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:39,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:40,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:41,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:41,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:42,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:42,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:50:42,965 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:50:43,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:44,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:45,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:46,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:46,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:46,682 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:50:47,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:47,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:50:47,248 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:50:47,249 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:50:47,251 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:50:47,391 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:50:48,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:48,392 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:50:49,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:50,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:51,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:52,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:52,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:53,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:54,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:55,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:56,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:57,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:57,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:50:57,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:50:57,963 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:50:58,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:50:59,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:00,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:01,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:02,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:02,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:03,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:04,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:04,401 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:51:05,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:06,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:07,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:08,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:08,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:09,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:10,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:11,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:12,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:12,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:51:12,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:51:13,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:14,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:14,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:15,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:16,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:16,689 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:51:17,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:18,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:19,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:19,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:20,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:21,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:21,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:51:21,185 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:51:21,185 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:51:21,186 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:51:21,435 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:51:22,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:22,436 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:51:23,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:24,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:25,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:25,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:26,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:27,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:27,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:51:27,963 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:51:28,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:29,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:30,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:30,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:31,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:32,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:33,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:34,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:34,444 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:51:35,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:35,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:36,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:37,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:38,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:39,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:40,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:41,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:41,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:42,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:42,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:51:42,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:51:43,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:44,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:45,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:46,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:46,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:46,697 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:51:47,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:48,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:49,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:50,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:51,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:51,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:51:51,455 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:51:51,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:51:51,457 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:51:51,465 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:51:51,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:52,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:52,466 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:51:53,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:54,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:55,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:56,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:57,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:57,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:51:57,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:51:57,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:51:58,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:51:59,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:00,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:01,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:02,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:02,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:03,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:04,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:05,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:06,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:06,476 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:52:07,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:08,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:08,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:09,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:10,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:11,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:12,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:12,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:52:12,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:52:13,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:13,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:14,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:15,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:16,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:16,703 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:52:17,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:18,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:18,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:19,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:20,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:21,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:21,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:52:21,702 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:52:21,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:52:21,703 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:52:22,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:22,493 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:52:22,493 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:52:23,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:23,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:24,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:24,494 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:52:25,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:26,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:27,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:27,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:52:27,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:52:28,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:28,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:29,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:30,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:31,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:32,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:33,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:34,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:34,501 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:52:34,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:35,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:36,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:37,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:38,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:39,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:39,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:40,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:41,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:42,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:42,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:52:42,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:52:43,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:44,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:44,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:45,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:46,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:46,703 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:52:47,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:48,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:49,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:49,270 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:52:49,271 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:52:49,271 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:52:49,275 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:52:49,519 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:52:49,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:50,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:50,520 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:52:51,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:52,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:53,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:54,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:55,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:55,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:52:56,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:57,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:57,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:52:57,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:52:58,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:52:59,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:00,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:00,526 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:53:00,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:01,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:02,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:03,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:04,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:05,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:06,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:06,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:07,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:08,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:09,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:10,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:11,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:11,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:12,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:12,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:53:12,973 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:53:13,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:14,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:15,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:16,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:53:16,128 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:53:16,128 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:53:16,130 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:53:16,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:16,547 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:53:16,707 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:53:16,709 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:17,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:18,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:18,549 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:53:19,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:20,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:21,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:22,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:22,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:23,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:24,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:25,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:26,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:27,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:27,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:53:27,968 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:53:28,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:28,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:29,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:30,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:30,556 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:53:31,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:32,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:33,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:33,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:34,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:35,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:36,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:37,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:38,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:38,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:39,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:40,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:41,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:42,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:42,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:53:42,977 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:53:43,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:43,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:44,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:45,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:46,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:46,710 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:53:46,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:53:46,879 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:53:46,879 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:53:46,880 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:53:47,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:47,578 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:53:48,301 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:48,578 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:53:49,301 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:49,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:50,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:51,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:52,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:53,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:54,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:55,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:55,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:53:56,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:57,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:57,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:53:57,974 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:53:58,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:53:59,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:00,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:00,587 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:54:00,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:01,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:02,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:03,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:04,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:05,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:06,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:06,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:07,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:08,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:09,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:10,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:11,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:11,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:12,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:12,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:54:12,978 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:54:13,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:14,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:15,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:16,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:16,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:16,714 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:54:17,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:54:17,211 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:54:17,211 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:54:17,212 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:54:17,606 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:54:17,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:18,608 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:54:18,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:19,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:20,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:21,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:22,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:22,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:23,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:24,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:25,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:26,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:27,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:27,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:54:27,974 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:54:28,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:28,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:29,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:30,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:31,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:32,618 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:54:32,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:33,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:33,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:34,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:35,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:36,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:37,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:38,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:39,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:39,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:40,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:41,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:42,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:42,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:54:42,983 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:54:43,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:44,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:44,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:45,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:46,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:46,724 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:54:47,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:48,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:54:48,237 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:54:48,237 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:54:48,239 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:54:48,634 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:54:48,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:49,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:49,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:50,636 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:54:50,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:51,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:52,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:53,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:54,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:55,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:54:55,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:56,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:57,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:57,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:54:57,980 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:54:58,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:54:59,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:00,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:01,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:01,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:02,644 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:55:02,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:03,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:04,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:05,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:06,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:06,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:07,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:08,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:09,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:10,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:11,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:11,692 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:12,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:12,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:55:12,989 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:55:13,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:14,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:15,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:16,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:16,723 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:55:16,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:17,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:55:17,540 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:55:17,541 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:55:17,543 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:55:17,669 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:55:17,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:18,670 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:55:18,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:19,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:20,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:21,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:22,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:22,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:23,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:24,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:25,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:26,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:27,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:27,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:55:27,984 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:55:28,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:28,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:29,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:30,677 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:55:30,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:31,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:32,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:33,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:33,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:34,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:35,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:36,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:37,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:38,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:38,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:39,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:40,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:41,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:42,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:42,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:55:42,986 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:55:43,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:43,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:44,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:45,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:46,725 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:55:46,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:55:46,818 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:55:46,818 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:55:46,820 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:55:47,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:47,695 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:55:48,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:48,697 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:55:49,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:49,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:50,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:51,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:52,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:53,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:54,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:54,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:55:55,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:56,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:57,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:57,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:55:57,986 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:55:58,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:59,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:55:59,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:00,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:01,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:02,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:02,706 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:56:03,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:04,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:05,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:05,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:06,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:07,451 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:08,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:09,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:10,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:11,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:11,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:12,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:12,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:56:12,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:56:13,458 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:14,458 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:15,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:16,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:16,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:16,729 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:56:17,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:18,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:56:18,407 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:56:18,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:56:18,408 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:56:18,723 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:56:18,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:19,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:20,725 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:56:20,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:21,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:21,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:22,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:23,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:24,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:25,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:26,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:27,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:27,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:27,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:56:27,988 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:56:28,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:30,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:30,732 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:56:31,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:32,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:33,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:33,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:34,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:35,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:36,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:37,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:38,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:38,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:39,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:40,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:41,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:42,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:42,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:56:42,999 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:56:43,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:43,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:44,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:44,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:56:44,957 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:56:44,958 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:56:44,960 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:56:45,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:45,762 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:56:46,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:46,732 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:56:46,763 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:56:47,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:48,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:49,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:49,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:50,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:51,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:52,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:53,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:54,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:55,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:55,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:56:56,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:57,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:57,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:56:57,992 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:56:58,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:56:58,770 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:56:59,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:00,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:00,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:01,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:02,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:03,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:04,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:05,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:06,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:06,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:07,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:08,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:09,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:10,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:11,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:11,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:12,102 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:12,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:57:12,993 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:57:13,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:14,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:57:14,031 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:57:14,031 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:57:14,033 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:57:14,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:14,791 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:57:14,791 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:57:15,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:16,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:16,737 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:57:16,792 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:57:17,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:17,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:18,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:19,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:20,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:21,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:22,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:22,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:23,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:24,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:25,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:26,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:26,800 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:57:27,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:27,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:27,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:57:27,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:57:28,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:29,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:30,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:31,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:32,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:33,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:33,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:34,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:35,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:36,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:37,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:38,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:38,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:39,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:40,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:41,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:41,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:57:41,344 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:57:41,344 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:57:41,345 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:57:41,825 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:57:42,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:42,826 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:57:42,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:57:42,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:57:43,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:44,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:44,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:45,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:46,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:46,740 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:57:47,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:48,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:49,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:50,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:50,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:51,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:52,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:53,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:54,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:55,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:56,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:56,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:57:56,836 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:57:57,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:57,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:57:57,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:57:58,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:57:59,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:00,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:01,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:01,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:02,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:03,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:04,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:05,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:06,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:07,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:07,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:08,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:09,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:10,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:11,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:12,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:12,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:12,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:58:12,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:58:13,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:13,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:58:13,349 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:58:13,350 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:58:13,351 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:58:13,854 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:58:14,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:14,855 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:58:15,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:16,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:16,745 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:58:17,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:18,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:18,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:19,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:20,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:21,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:22,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:23,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:23,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:24,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:25,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:26,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:26,863 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:58:27,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:27,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:58:27,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:58:28,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:28,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:29,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:30,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:31,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:32,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:33,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:34,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:34,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:35,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:36,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:37,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:38,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:39,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:39,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:40,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:41,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:42,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:58:42,151 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:58:42,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:58:42,153 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:58:42,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:42,883 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:58:42,884 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:58:42,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:58:42,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:58:43,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:44,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:44,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:44,885 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:58:45,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:46,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:46,748 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:58:47,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:48,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:49,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:50,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:50,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:51,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:52,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:53,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:54,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:55,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:56,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:58:56,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:56,894 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:58:57,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:57,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:58:57,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:58:58,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:58:59,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:00,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:01,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:01,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:02,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:03,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:04,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:05,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:06,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:06,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:07,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:08,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:09,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:10,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:11,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:11,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:12,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:13,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:59:13,003 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:59:13,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:14,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:14,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:59:14,892 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:59:14,892 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:59:14,894 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:59:14,916 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:59:15,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:16,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:16,752 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:59:16,918 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:59:17,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:17,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:18,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:19,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:20,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:21,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:22,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:22,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:23,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:24,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:25,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:26,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:27,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:27,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:59:27,999 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:59:28,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:28,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:28,927 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:59:29,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:30,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:31,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:32,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:33,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:33,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:34,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:35,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:36,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:37,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:38,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:38,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:39,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:40,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:41,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:42,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:43,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:59:43,002 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:59:43,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:44,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 11:59:44,025 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 11:59:44,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 11:59:44,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:44,027 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 11:59:44,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:44,952 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 11:59:44,953 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 11:59:45,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:46,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:46,757 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 11:59:47,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:48,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:49,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:49,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:50,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:51,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:52,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:53,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:54,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:55,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:55,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 11:59:56,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:57,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:58,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 11:59:58,002 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 11:59:58,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 11:59:59,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:00,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:00,963 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:00:01,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:01,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:02,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:03,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:04,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:05,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:06,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:06,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:07,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:08,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:09,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:10,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:11,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:12,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:12,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:13,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:00:13,002 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:00:13,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:14,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:15,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:16,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:16,761 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:00:17,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:18,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:18,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:19,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:00:19,441 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:00:19,441 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:00:19,443 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:00:19,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:19,984 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:00:20,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:20,985 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:00:21,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:22,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:23,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:23,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:24,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:25,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:26,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:27,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:28,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:00:28,002 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:00:28,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:29,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:29,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:30,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:31,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:32,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:33,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:34,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:34,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:34,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:00:35,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:36,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:37,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:38,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:39,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:40,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:40,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:41,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:42,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:43,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:00:43,002 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:00:43,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:44,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:45,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:46,776 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:00:46,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:46,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:47,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:48,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:49,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:50,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:51,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:00:51,750 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:00:51,750 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:00:51,752 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:00:51,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:52,016 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:00:52,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:53,017 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:00:53,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:54,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:55,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:56,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:56,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:00:57,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:58,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:00:58,003 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:00:58,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:00:59,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:00,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:01,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:02,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:02,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:03,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:04,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:05,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:06,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:07,028 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:01:07,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:07,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:08,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:09,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:10,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:11,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:12,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:13,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:01:13,003 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:01:13,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:13,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:14,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:15,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:16,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:16,782 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:01:17,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:18,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:18,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:19,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:20,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:21,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:22,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:22,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:01:22,393 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:01:22,394 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:01:22,395 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:01:23,048 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:01:23,048 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:01:23,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:23,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:24,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:25,049 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:01:25,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:26,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:27,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:28,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:01:28,003 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:01:28,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:28,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:29,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:30,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:31,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:32,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:33,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:33,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:34,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:35,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:36,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:37,057 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:01:37,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:38,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:38,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:39,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:40,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:41,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:42,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:43,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:01:43,003 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:01:43,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:44,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:44,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:45,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:46,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:46,774 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:01:47,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:48,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:49,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:49,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:50,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:51,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:52,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:53,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:53,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:01:53,368 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:01:53,369 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:01:53,374 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:01:54,079 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:01:54,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:55,080 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:01:55,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:55,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:01:56,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:57,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:58,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:01:58,003 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:01:58,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:01:59,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:00,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:01,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:01,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:02,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:03,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:04,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:05,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:06,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:07,089 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:02:07,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:07,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:08,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:09,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:10,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:11,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:12,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:12,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:13,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:02:13,003 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:02:13,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:14,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:15,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:16,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:16,787 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:02:17,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:18,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:18,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:19,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:20,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:21,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:22,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:22,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:02:22,429 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:02:22,429 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:02:22,430 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:02:23,109 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:02:23,110 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:02:23,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:23,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:24,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:25,111 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:02:25,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:26,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:27,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:28,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:02:28,004 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:02:28,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:28,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:29,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:30,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:31,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:32,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:33,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:33,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:34,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:35,117 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:02:35,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:36,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:37,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:38,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:38,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:39,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:40,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:41,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:42,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:43,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:02:43,004 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:02:43,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:43,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:44,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:45,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:46,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:46,792 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:02:47,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:48,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:48,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:49,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:50,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:50,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:02:50,658 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:02:50,659 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:02:50,660 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:02:51,134 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:02:51,134 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:02:51,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:52,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:53,135 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:02:53,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:53,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:54,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:55,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:56,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:57,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:58,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:02:58,004 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:02:58,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:02:58,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:02:59,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:00,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:01,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:02,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:03,142 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:03:03,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:03,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:04,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:05,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:06,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:07,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:08,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:09,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:09,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:10,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:11,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:12,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:13,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:03:13,004 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:03:13,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:14,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:14,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:15,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:16,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:16,784 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:03:17,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:18,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:19,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:19,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:03:19,588 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:03:19,589 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:03:19,590 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:03:19,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:20,161 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:03:20,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:21,162 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:03:21,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:22,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:23,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:24,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:25,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:25,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:26,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:27,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:28,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:03:28,004 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:03:28,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:29,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:30,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:31,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:31,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:32,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:33,170 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:03:33,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:34,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:35,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:36,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:36,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:37,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:38,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:39,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:40,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:41,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:42,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:42,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:43,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:03:43,016 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:03:43,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:44,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:45,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:46,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:46,796 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:03:47,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:48,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:48,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:03:48,406 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:03:48,407 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:03:48,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:48,408 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:03:49,187 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:03:49,188 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:03:49,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:50,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:51,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:52,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:53,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:53,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:03:54,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:55,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:56,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:57,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:58,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:03:58,013 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:03:58,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:59,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:03:59,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:00,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:01,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:02,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:03,198 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:04:03,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:04,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:04,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:05,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:06,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:07,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:08,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:09,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:10,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:10,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:11,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:12,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:13,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:04:13,013 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:04:13,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:14,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:15,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:15,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:16,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:16,794 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:04:17,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:18,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:19,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:19,489 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:04:19,491 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:04:19,491 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:04:19,492 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:04:20,215 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:04:20,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:21,216 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:04:21,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:21,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:22,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:23,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:24,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:25,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:26,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:26,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:27,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:28,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:04:28,013 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:04:28,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:29,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:30,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:31,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:32,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:32,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:33,224 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:04:33,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:34,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:35,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:36,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:37,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:37,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:38,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:39,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:40,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:41,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:42,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:42,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:43,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:04:43,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:04:43,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:44,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:45,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:46,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:46,812 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:04:47,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:48,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:48,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:49,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:50,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:50,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:04:50,448 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:04:50,449 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:04:50,450 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:04:51,243 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:04:51,243 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:04:51,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:52,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:53,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:53,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:54,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:55,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:56,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:57,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:58,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:04:58,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:04:58,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:04:58,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:04:59,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:00,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:01,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:02,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:03,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:03,251 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:05:03,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:04,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:05,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:06,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:07,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:08,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:09,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:09,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:10,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:11,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:12,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:13,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:05:13,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:05:13,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:14,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:14,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:15,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:16,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:16,807 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:05:17,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:18,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:19,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:19,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:19,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:05:19,945 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:05:19,945 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:05:19,946 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:05:20,282 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:05:20,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:21,283 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:05:21,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:22,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:23,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:24,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:25,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:25,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:26,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:27,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:28,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:05:28,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:05:28,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:29,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:30,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:31,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:31,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:32,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:33,289 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:05:33,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:34,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:35,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:36,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:36,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:37,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:38,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:39,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:40,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:41,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:41,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:42,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:43,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:05:43,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:05:43,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:44,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:45,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:46,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:46,813 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:05:47,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:47,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:48,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:49,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:50,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:05:50,158 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:05:50,158 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:05:50,160 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:05:50,307 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:05:50,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:51,308 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:05:51,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:52,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:53,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:53,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:54,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:55,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:56,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:57,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:58,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:05:58,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:05:58,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:05:58,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:05:59,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:00,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:01,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:02,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:03,314 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:06:03,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:03,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:04,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:05,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:06,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:07,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:08,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:08,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:09,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:10,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:11,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:12,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:13,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:06:13,015 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:06:13,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:13,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:14,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:15,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:16,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:16,823 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:06:17,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:18,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:19,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:06:19,053 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:06:19,053 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:06:19,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:19,055 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:06:19,329 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:06:19,330 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:06:19,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:20,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:21,331 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:06:21,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:22,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:23,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:24,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:24,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:25,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:26,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:27,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:28,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:06:28,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:06:28,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:29,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:30,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:30,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:31,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:32,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:33,339 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:06:33,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:34,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:35,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:35,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:36,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:37,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:38,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:39,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:40,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:40,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:41,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:42,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:43,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:06:43,015 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:06:43,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:44,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:45,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:45,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:46,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:46,821 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:06:47,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:48,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:49,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:50,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:50,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:51,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:06:51,358 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:06:51,358 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:06:51,360 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:06:51,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:52,360 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:06:52,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:53,361 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:06:53,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:54,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:55,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:56,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:56,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:06:57,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:58,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:06:58,015 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:06:58,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:06:59,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:00,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:01,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:02,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:02,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:03,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:04,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:05,367 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:07:05,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:06,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:07,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:07,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:08,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:09,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:10,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:11,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:12,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:13,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:07:13,015 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:07:13,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:13,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:14,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:15,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:16,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:16,822 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:07:17,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:18,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:18,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:19,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:20,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:07:20,673 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:07:20,673 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:07:20,675 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:07:21,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:21,383 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:07:21,383 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:07:21,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:22,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:23,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:23,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:24,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:25,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:26,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:27,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:28,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:07:28,015 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:07:28,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:29,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:29,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:30,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:31,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:32,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:33,390 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:07:33,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:34,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:34,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:35,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:36,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:37,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:38,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:39,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:40,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:40,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:41,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:42,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:43,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:07:43,035 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:07:43,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:44,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:45,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:46,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:46,824 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:07:46,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:47,078 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:07:47,080 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:07:47,081 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:07:47,082 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:07:47,415 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:07:47,416 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:07:47,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:48,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:49,417 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:07:49,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:50,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:51,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:51,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:52,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:53,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:54,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:55,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:56,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:56,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:07:57,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:58,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:07:58,020 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:07:58,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:07:59,423 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:07:59,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:00,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:01,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:01,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:02,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:03,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:04,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:05,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:06,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:07,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:07,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:08,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:09,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:10,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:11,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:12,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:12,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:13,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:08:13,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:08:13,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:14,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:15,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:16,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:08:16,233 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:08:16,234 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:08:16,236 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:08:16,445 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:08:16,826 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:08:16,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:17,446 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:08:17,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:18,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:18,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:19,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:20,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:21,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:22,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:23,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:23,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:24,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:25,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:26,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:27,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:28,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:08:28,022 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:08:28,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:29,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:29,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:30,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:31,454 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:08:31,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:32,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:33,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:34,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:34,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:35,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:36,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:37,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:38,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:39,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:39,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:40,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:41,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:42,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:43,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:08:43,026 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:08:43,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:44,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:44,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:45,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:46,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:08:46,151 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:08:46,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:08:46,152 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:08:46,474 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:08:46,830 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:08:46,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:47,475 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:08:47,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:48,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:49,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:50,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:50,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:51,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:52,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:53,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:54,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:55,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:56,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:56,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:08:57,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:58,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:08:58,022 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:08:58,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:08:59,483 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:08:59,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:00,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:01,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:01,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:02,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:03,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:04,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:05,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:06,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:06,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:07,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:08,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:09,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:10,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:11,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:11,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:12,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:13,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:09:13,023 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:09:13,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:14,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:15,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:16,373 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:09:16,375 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:09:16,375 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:09:16,377 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:09:16,503 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:09:16,834 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:09:16,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:17,507 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:09:17,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:17,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:18,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:19,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:20,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:21,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:22,913 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:22,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:23,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:24,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:25,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:26,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:27,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:28,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:09:28,023 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:09:28,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:28,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:29,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:30,515 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:09:30,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:31,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:32,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:33,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:33,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:34,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:35,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:36,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:37,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:38,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:38,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:39,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:40,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:41,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:42,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:43,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:09:43,029 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:09:43,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:43,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:44,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:45,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:46,840 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:09:46,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:47,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:09:47,263 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:09:47,263 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:09:47,264 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:09:47,532 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:09:47,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:48,533 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:09:48,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:49,918 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:49,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:50,534 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:09:50,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:51,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:52,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:53,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:54,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:54,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:09:55,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:56,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:57,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:58,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:09:58,029 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:09:58,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:09:59,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:00,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:00,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:01,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:02,541 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:10:02,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:03,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:04,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:05,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:05,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:06,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:07,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:08,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:09,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:10,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:11,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:11,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:12,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:13,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:10:13,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:10:13,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:14,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:15,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:16,844 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:10:16,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:17,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:17,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:18,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:19,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:20,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:10:20,207 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:10:20,207 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:10:20,209 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:10:20,566 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:10:20,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:21,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:22,567 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:10:22,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:23,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:23,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:24,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:25,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:26,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:27,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:28,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:10:28,029 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:10:28,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:28,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:29,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:30,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:31,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:32,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:33,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:33,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:34,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:35,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:36,575 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:10:36,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:37,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:38,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:38,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:39,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:40,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:41,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:42,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:43,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:10:43,029 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:10:43,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:43,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:44,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:45,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:46,852 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:10:46,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:47,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:48,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:48,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:49,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:50,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:51,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:52,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:10:52,731 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:10:52,732 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:10:52,733 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:10:53,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:53,597 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:10:54,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:54,598 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:10:54,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:10:55,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:56,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:57,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:58,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:10:58,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:10:58,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:10:59,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:00,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:00,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:01,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:02,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:03,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:04,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:05,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:06,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:06,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:07,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:08,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:08,606 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:11:09,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:10,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:11,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:11,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:12,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:13,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:11:13,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:11:13,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:14,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:15,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:16,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:16,847 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:11:17,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:17,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:18,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:19,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:20,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:21,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:22,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:22,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:23,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:24,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:24,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:11:24,239 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:11:24,240 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:11:24,245 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:11:24,627 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:11:25,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:26,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:26,629 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:11:27,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:28,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:11:28,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:11:28,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:28,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:29,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:30,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:31,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:32,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:33,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:33,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:34,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:35,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:36,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:37,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:38,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:38,638 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:11:38,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:39,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:40,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:41,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:42,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:43,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:11:43,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:11:43,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:44,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:44,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:45,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:46,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:46,851 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:11:47,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:48,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:49,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:49,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:50,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:51,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:52,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:53,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:54,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:54,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:11:54,626 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:11:54,627 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:11:54,631 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:11:54,668 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:11:55,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:11:55,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:56,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:56,670 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:11:57,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:58,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:11:58,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:11:58,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:11:59,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:00,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:00,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:01,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:02,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:03,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:04,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:05,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:06,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:06,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:07,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:08,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:09,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:10,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:10,678 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:12:11,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:11,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:12,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:13,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:12:13,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:12:13,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:14,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:15,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:16,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:16,855 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:12:17,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:17,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:18,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:19,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:20,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:21,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:22,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:22,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:23,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:24,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:25,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:26,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:27,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:12:27,004 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:12:27,004 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:12:27,008 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:12:27,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:27,697 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:12:28,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:12:28,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:12:28,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:28,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:28,698 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:12:29,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:30,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:31,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:32,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:33,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:33,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:34,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:35,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:36,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:37,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:38,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:39,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:39,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:40,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:41,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:42,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:43,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:12:43,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:12:43,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:44,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:44,708 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:12:44,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:45,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:46,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:46,860 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:12:47,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:48,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:49,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:49,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:50,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:51,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:52,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:53,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:54,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:55,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:12:55,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:56,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:57,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:58,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:12:58,041 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:12:58,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:59,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:12:59,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:12:59,701 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:12:59,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:12:59,705 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:12:59,735 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:13:00,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:00,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:01,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:02,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:02,737 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:13:03,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:04,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:05,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:05,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:06,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:07,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:08,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:09,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:10,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:11,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:11,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:12,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:13,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:13:13,033 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:13:13,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:14,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:14,746 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:13:15,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:16,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:16,863 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:13:17,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:17,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:18,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:19,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:20,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:21,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:22,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:22,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:23,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:24,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:25,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:26,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:27,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:13:27,538 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:13:27,538 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:13:27,539 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:13:27,767 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:13:27,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:28,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:13:28,033 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:13:28,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:28,768 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:13:28,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:29,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:30,770 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:13:30,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:31,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:32,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:33,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:34,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:34,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:35,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:36,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:37,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:38,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:39,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:39,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:40,775 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:13:40,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:41,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:42,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:43,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:13:43,034 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:13:43,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:44,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:45,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:45,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:46,866 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:13:46,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:47,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:48,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:49,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:50,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:51,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:51,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:52,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:53,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:54,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:55,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:56,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:13:56,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:57,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:13:57,063 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:13:57,064 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:13:57,066 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:13:57,801 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:13:57,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:58,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:13:58,034 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:13:58,801 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:13:58,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:13:59,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:00,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:01,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:02,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:02,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:03,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:04,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:05,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:06,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:07,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:07,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:08,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:09,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:10,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:12,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:12,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:13,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:14:13,034 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:14:13,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:14,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:14,811 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:14:15,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:16,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:16,870 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:14:17,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:17,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:18,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:19,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:20,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:21,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:22,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:23,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:23,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:24,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:25,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:26,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:27,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:28,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:28,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:14:28,039 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:14:28,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:29,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:30,097 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:31,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:14:31,026 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:14:31,027 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:14:31,028 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:14:31,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:31,827 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:14:32,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:32,827 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:14:33,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:33,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:34,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:35,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:36,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:37,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:38,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:39,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:39,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:40,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:41,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:42,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:43,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:14:43,036 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:14:43,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:44,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:45,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:45,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:46,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:46,875 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:14:47,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:48,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:48,837 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:14:49,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:50,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:50,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:51,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:52,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:53,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:54,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:55,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:14:55,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:56,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:57,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:58,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:14:58,036 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:14:58,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:14:59,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:00,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:00,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:01,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:02,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:03,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:04,176 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:04,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:15:04,857 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:15:04,857 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:15:04,859 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:15:05,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:05,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:05,854 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:15:06,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:06,855 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:15:07,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:08,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:09,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:10,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:11,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:11,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:12,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:13,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:15:13,036 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:15:13,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:14,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:15,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:16,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:16,878 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:15:16,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:17,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:18,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:18,861 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:15:19,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:20,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:21,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:22,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:22,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:23,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:24,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:25,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:26,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:27,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:28,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:28,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:15:28,036 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:15:28,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:29,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:30,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:31,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:32,270 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:33,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:33,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:34,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:35,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:35,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:15:35,298 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:15:35,298 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:15:35,300 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:15:35,886 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:15:36,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:36,886 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:15:37,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:38,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:38,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:39,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:40,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:41,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:42,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:43,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:15:43,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:15:43,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:44,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:44,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:45,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:46,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:46,881 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:15:47,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:48,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:49,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:49,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:50,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:50,896 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:15:51,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:52,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:53,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:54,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:54,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:55,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:56,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:57,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:58,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:15:58,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:15:58,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:15:59,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:15:59,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:00,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:01,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:02,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:03,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:04,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:04,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:05,399 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:06,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:07,401 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:07,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:16:07,989 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:16:07,989 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:16:07,990 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:16:08,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:08,916 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:16:09,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:10,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:10,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:10,917 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:16:11,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:12,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:13,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:16:13,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:16:13,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:14,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:15,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:16,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:16,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:16,885 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:16:17,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:18,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:19,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:20,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:21,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:22,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:22,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:23,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:24,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:24,926 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:16:25,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:26,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:27,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:27,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:28,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:16:28,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:16:28,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:29,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:30,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:31,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:32,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:33,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:33,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:34,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:35,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:36,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:37,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:38,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:38,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:39,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:40,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:41,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:16:41,013 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:16:41,013 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:16:41,014 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:16:41,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:41,947 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:16:42,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:42,948 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:16:43,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:16:43,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:16:43,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:43,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:44,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:45,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:46,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:46,890 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:16:47,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:48,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:48,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:49,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:50,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:51,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:52,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:53,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:54,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:16:54,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:55,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:56,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:56,957 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:16:57,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:58,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:16:58,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:16:58,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:16:59,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:00,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:00,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:01,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:02,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:03,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:04,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:05,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:05,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:06,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:07,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:08,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:09,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:10,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:10,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:11,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:12,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:12,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:17:12,897 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:17:12,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:17:12,899 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:17:12,986 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:17:13,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:17:13,310 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:17:13,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:14,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:14,987 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:17:15,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:15,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:16,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:16,894 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:17:17,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:18,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:19,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:20,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:21,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:21,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:22,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:23,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:24,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:25,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:26,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:27,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:27,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:28,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:17:28,040 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:17:28,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:28,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:17:29,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:30,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:31,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:32,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:32,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:33,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:34,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:35,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:36,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:37,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:38,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:38,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:39,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:40,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:41,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:42,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:43,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:43,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:17:43,046 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:17:43,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:44,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:45,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:17:45,567 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:17:45,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:17:45,569 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:17:45,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:46,016 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:17:46,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:46,898 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:17:47,017 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:17:47,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:48,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:48,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:49,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:50,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:51,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:52,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:53,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:54,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:54,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:55,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:56,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:57,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:58,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:17:58,246 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:17:58,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:17:59,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:17:59,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:00,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:01,026 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:18:01,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:02,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:03,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:04,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:04,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:05,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:06,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:07,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:08,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:09,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:10,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:10,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:11,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:12,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:13,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:18:13,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:18:13,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:14,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:15,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:15,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:15,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:18:15,889 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:18:15,890 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:18:15,891 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:18:16,041 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:18:16,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:16,902 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:18:17,042 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:18:17,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:18,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:19,043 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:18:19,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:20,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:21,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:21,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:22,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:23,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:24,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:25,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:26,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:26,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:27,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:28,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:18:28,371 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:18:28,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:29,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:30,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:31,051 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:18:31,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:32,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:32,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:33,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:34,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:35,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:36,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:37,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:37,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:38,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:39,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:40,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:41,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:42,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:42,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:43,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:18:43,210 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:18:43,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:44,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:45,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:45,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:18:45,864 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:18:45,864 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:18:45,866 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:18:46,080 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:18:46,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:46,906 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:18:47,081 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:18:47,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:47,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:48,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:49,082 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:18:49,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:50,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:51,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:52,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:53,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:53,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:54,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:55,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:56,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:57,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:58,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:18:58,390 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:18:58,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:18:58,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:18:59,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:00,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:01,090 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:19:01,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:02,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:03,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:04,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:04,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:05,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:06,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:07,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:08,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:09,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:10,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:10,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:11,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:12,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:13,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:19:13,240 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:19:13,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:14,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:15,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:15,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:16,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:19:16,449 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:19:16,449 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:19:16,450 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:19:16,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:16,910 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:19:17,110 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:19:17,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:18,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:19,112 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:19:19,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:20,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:21,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:21,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:22,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:23,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:24,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:25,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:26,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:26,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:27,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:28,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:19:28,235 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:19:28,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:29,121 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:19:29,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:30,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:31,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:31,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:32,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:33,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:34,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:35,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:36,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:37,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:37,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:38,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:39,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:40,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:41,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:42,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:42,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:43,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:19:43,240 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:19:43,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:44,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:19:44,589 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:19:44,589 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:19:44,591 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:19:45,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:45,148 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:19:46,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:46,914 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:19:47,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:47,150 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:19:47,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:48,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:49,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:50,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:51,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:52,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:53,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:53,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:54,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:55,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:56,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:57,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:58,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:58,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:19:58,237 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:19:58,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:19:59,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:19:59,157 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:20:00,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:01,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:02,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:03,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:04,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:04,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:05,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:06,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:07,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:08,005 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:09,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:09,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:10,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:11,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:12,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:13,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:13,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:20:13,237 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:20:14,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:14,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:14,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:20:14,204 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:20:14,205 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:20:14,207 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:20:15,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:15,176 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:20:16,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:16,919 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:20:17,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:17,177 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:20:18,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:19,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:19,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:20,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:21,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:22,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:23,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:24,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:25,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:25,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:26,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:27,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:28,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:28,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:20:28,237 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:20:29,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:29,185 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:20:30,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:30,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:31,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:32,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:33,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:34,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:35,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:36,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:36,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:37,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:38,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:39,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:40,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:41,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:41,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:42,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:43,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:43,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:20:43,237 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:20:44,009 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:20:44,011 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:20:44,011 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:20:44,012 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:20:44,204 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:20:44,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:45,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:46,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:46,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:46,922 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:20:47,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:47,207 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:20:48,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:49,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:50,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:51,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:52,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:52,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:53,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:54,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:55,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:56,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:57,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:57,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:20:58,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:58,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:20:58,238 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:20:59,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:20:59,215 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:21:00,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:01,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:02,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:03,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:03,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:04,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:05,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:06,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:07,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:08,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:09,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:09,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:10,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:11,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:12,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:13,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:13,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:21:13,243 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:21:14,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:15,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:21:15,025 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:21:15,026 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:21:15,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:15,027 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:21:15,239 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:21:15,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:16,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:16,926 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:21:17,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:17,240 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:21:18,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:19,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:20,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:20,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:21,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:22,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:23,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:24,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:25,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:26,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:26,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:27,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:28,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:28,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:21:28,243 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:21:29,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:29,250 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:21:30,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:31,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:31,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:32,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:33,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:34,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:35,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:36,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:37,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:37,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:38,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:39,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:40,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:41,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:42,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:42,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:43,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:43,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:21:43,251 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:21:44,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:44,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:21:44,349 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:21:44,349 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:21:44,351 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:21:45,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:45,277 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:21:46,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:46,930 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:21:47,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:47,279 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:21:47,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:48,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:49,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:50,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:51,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:52,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:53,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:53,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:21:54,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:55,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:56,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:57,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:58,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:58,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:21:58,244 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:21:59,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:21:59,287 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:21:59,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:00,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:01,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:02,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:03,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:04,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:04,491 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:05,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:06,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:07,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:08,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:09,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:09,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:10,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:11,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:12,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:13,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:13,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:22:13,245 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:22:13,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:22:13,933 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:22:13,934 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:22:13,935 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:22:14,309 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:22:14,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:15,310 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:22:15,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:15,493 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:16,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:16,934 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:22:17,312 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:22:17,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:18,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:19,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:20,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:20,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:21,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:22,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:23,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:24,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:25,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:25,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:26,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:27,318 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:22:27,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:28,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:22:28,245 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:22:28,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:29,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:30,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:31,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:31,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:32,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:33,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:34,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:35,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:36,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:37,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:37,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:38,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:39,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:40,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:41,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:42,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:42,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:43,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:22:43,089 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:22:43,089 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:22:43,090 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:22:43,348 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:22:43,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:43,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:22:43,505 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:22:44,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:45,350 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:22:45,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:46,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:46,939 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:22:47,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:47,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:48,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:49,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:50,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:51,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:52,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:53,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:53,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:54,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:55,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:56,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:57,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:58,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:22:58,503 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:22:58,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:22:58,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:22:59,357 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:22:59,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:00,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:01,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:02,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:03,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:04,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:04,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:05,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:06,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:07,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:08,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:09,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:09,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:10,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:11,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:12,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:13,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:23:13,503 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:23:13,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:13,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:23:13,708 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:23:13,709 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:23:13,710 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:23:14,380 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:23:14,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:15,380 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:23:15,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:15,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:16,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:16,942 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:23:17,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:18,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:19,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:20,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:21,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:21,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:22,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:23,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:24,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:25,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:26,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:26,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:27,390 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:23:27,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:28,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:23:28,503 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:23:28,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:29,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:30,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:31,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:32,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:32,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:33,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:34,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:35,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:36,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:37,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:37,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:38,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:39,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:40,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:41,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:42,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:42,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:43,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:23:43,233 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:23:43,233 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:23:43,235 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:23:43,420 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:23:43,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:43,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:23:43,653 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:23:44,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:45,422 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:23:45,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:46,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:46,946 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:23:47,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:47,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:48,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:49,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:50,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:51,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:52,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:53,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:53,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:54,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:55,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:56,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:57,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:58,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:23:58,649 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:23:58,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:23:58,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:23:59,429 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:23:59,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:00,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:01,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:02,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:03,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:04,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:04,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:05,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:06,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:07,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:08,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:09,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:09,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:10,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:11,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:12,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:13,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:24:13,665 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:24:13,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:14,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:14,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:15,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:15,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:24:15,885 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:24:15,886 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:24:15,888 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:24:16,447 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:24:16,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:16,950 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:24:17,448 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:24:17,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:18,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:19,450 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:24:19,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:20,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:20,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:21,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:22,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:23,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:24,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:25,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:26,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:26,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:27,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:28,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:24:28,653 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:24:28,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:29,457 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:24:29,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:30,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:31,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:31,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:32,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:33,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:34,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:35,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:36,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:37,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:37,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:38,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:39,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:40,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:41,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:42,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:42,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:43,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:24:43,658 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:24:43,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:44,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:24:44,429 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:24:44,430 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:24:44,432 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:24:44,484 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:24:44,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:45,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:46,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:46,954 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:24:47,486 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:24:47,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:47,956 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:48,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:49,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:50,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:51,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:52,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:53,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:53,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:54,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:55,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:56,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:57,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:24:58,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:24:59,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:24:59,063 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:24:59,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:00,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:01,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:01,495 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:25:02,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:03,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:04,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:04,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:05,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:06,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:07,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:08,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:09,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:10,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:10,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:11,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:12,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:13,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:14,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:25:14,066 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:25:14,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:15,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:15,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:16,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:16,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:25:16,463 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:25:16,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:25:16,465 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:25:16,519 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:25:16,956 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:25:17,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:18,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:19,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:19,521 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:25:20,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:21,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:21,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:22,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:23,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:24,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:25,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:26,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:27,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:27,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:28,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:29,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:25:29,193 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:25:29,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:30,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:31,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:31,527 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:25:32,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:32,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:33,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:34,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:35,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:36,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:37,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:38,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:38,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:39,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:40,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:41,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:42,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:43,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:43,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:44,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:25:44,196 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:25:44,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:45,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:46,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:46,964 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:25:47,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:47,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:25:47,650 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:25:47,651 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:25:47,652 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:25:48,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:48,548 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:25:49,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:49,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:49,548 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:25:50,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:51,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:52,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:53,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:54,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:54,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:25:55,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:56,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:57,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:58,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:59,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:25:59,191 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:25:59,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:25:59,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:00,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:01,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:01,556 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:26:02,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:03,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:04,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:04,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:05,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:06,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:07,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:08,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:09,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:10,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:10,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:11,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:12,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:13,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:14,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:26:14,195 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:26:14,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:15,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:15,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:16,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:16,965 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:26:17,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:26:17,083 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:26:17,084 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:26:17,086 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:26:17,491 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:17,572 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:26:18,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:19,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:19,574 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:26:20,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:21,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:21,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:22,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:23,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:24,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:25,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:26,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:27,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:27,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:28,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:29,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:26:29,196 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:26:29,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:30,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:31,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:31,582 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:26:32,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:32,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:33,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:34,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:35,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:36,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:37,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:37,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:38,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:39,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:40,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:41,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:42,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:42,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:43,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:44,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:26:44,196 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:26:44,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:45,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:46,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:46,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:26:46,427 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:26:46,428 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:26:46,429 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:26:46,603 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:26:46,969 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:26:47,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:47,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:48,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:49,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:49,606 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:26:50,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:51,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:52,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:53,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:53,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:54,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:55,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:56,236 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:57,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:58,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:26:58,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:26:59,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:26:59,196 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:26:59,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:00,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:01,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:02,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:03,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:03,616 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:27:03,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:04,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:05,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:06,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:07,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:08,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:09,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:09,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:10,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:11,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:12,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:13,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:14,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:27:14,199 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:27:14,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:14,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:15,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:16,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:16,972 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:27:17,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:18,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:27:18,190 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:27:18,190 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:27:18,192 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:27:18,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:18,633 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:27:19,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:19,634 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:27:20,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:20,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:21,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:21,636 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:27:22,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:23,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:24,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:25,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:25,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:26,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:27,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:28,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:29,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:27:29,199 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:27:29,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:30,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:31,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:31,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:32,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:33,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:33,645 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:27:34,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:35,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:36,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:36,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:37,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:38,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:39,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:40,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:41,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:42,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:42,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:43,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:44,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:27:44,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:27:44,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:45,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:46,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:46,975 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:27:47,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:47,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:48,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:48,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:27:48,845 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:27:48,845 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:27:48,846 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:27:49,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:49,672 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:27:50,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:51,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:51,674 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:27:52,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:53,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:53,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:54,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:55,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:56,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:57,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:58,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:27:58,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:27:59,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:27:59,202 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:27:59,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:00,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:01,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:02,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:03,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:03,682 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:28:03,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:04,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:05,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:06,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:07,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:08,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:09,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:09,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:10,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:11,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:12,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:13,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:14,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:28:14,213 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:28:14,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:14,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:15,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:16,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:16,980 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:28:17,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:18,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:18,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:28:18,649 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:28:18,649 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:28:18,651 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:28:18,702 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:28:19,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:19,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:20,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:21,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:21,704 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:28:22,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:23,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:24,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:25,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:25,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:26,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:27,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:28,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:29,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:28:29,207 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:28:29,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:30,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:31,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:31,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:32,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:33,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:33,712 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:28:34,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:35,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:36,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:36,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:37,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:38,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:39,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:40,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:41,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:42,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:42,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:43,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:44,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:28:44,207 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:28:44,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:45,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:46,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:46,987 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:28:47,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:47,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:48,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:48,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:28:48,902 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:28:48,902 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:28:48,903 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:28:49,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:49,732 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:28:50,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:51,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:51,733 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:28:52,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:53,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:53,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:54,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:55,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:56,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:57,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:58,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:28:58,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:28:59,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:28:59,207 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:28:59,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:00,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:01,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:02,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:03,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:04,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:04,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:05,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:05,743 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:29:06,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:07,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:08,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:09,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:09,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:10,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:11,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:12,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:13,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:14,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:29:14,207 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:29:14,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:15,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:15,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:16,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:16,990 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:29:17,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:18,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:19,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:20,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:20,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:21,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:29:21,275 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:29:21,275 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:29:21,277 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:29:21,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:21,761 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:29:22,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:23,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:23,762 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:29:24,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:25,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:25,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:26,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:27,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:28,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:29,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:29:29,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:29:29,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:30,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:31,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:31,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:32,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:33,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:34,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:35,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:36,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:37,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:37,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:37,771 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:29:38,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:39,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:40,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:41,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:42,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:42,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:43,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:44,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:29:44,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:29:44,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:45,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:46,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:46,989 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:29:47,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:47,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:48,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:49,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:50,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:51,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:52,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:53,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:53,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:53,969 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:29:53,971 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:29:53,971 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:29:53,976 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:29:54,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:54,791 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:29:55,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:55,792 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:29:56,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:57,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:58,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:29:59,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:29:59,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:29:59,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:29:59,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:00,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:01,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:02,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:03,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:04,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:04,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:05,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:06,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:07,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:07,801 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:30:08,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:09,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:09,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:10,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:11,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:12,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:13,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:14,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:30:14,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:30:14,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:15,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:15,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:16,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:16,993 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:30:17,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:18,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:19,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:20,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:20,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:21,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:22,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:23,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:30:23,694 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:30:23,695 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:30:23,697 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:30:23,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:23,820 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:30:24,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:25,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:25,822 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:30:26,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:26,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:27,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:28,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:29,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:30:29,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:30:29,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:30,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:31,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:32,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:32,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:33,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:34,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:35,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:36,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:37,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:37,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:38,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:39,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:39,832 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:30:40,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:41,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:42,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:43,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:43,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:44,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:30:44,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:30:44,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:45,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:46,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:46,998 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:30:47,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:48,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:49,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:49,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:50,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:51,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:52,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:53,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:54,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:54,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:55,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:55,986 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:30:55,987 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:30:55,987 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:30:55,991 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:30:56,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:56,848 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:30:57,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:57,849 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:30:58,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:30:59,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:30:59,208 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:30:59,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:30:59,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:00,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:01,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:02,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:03,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:04,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:04,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:05,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:06,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:07,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:08,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:09,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:09,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:10,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:11,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:11,858 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:31:12,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:13,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:14,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:31:14,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:31:14,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:14,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:15,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:16,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:17,002 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:31:17,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:18,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:19,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:20,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:20,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:21,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:22,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:23,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:24,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:25,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:26,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:26,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:27,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:28,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:31:28,042 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:31:28,043 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:31:28,048 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:31:28,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:28,882 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:31:29,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:31:29,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:31:29,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:29,883 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:31:30,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:31,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:32,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:32,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:33,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:34,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:35,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:36,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:37,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:37,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:38,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:39,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:40,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:41,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:41,891 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:31:42,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:43,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:43,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:44,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:31:44,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:31:44,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:45,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:46,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:47,006 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:31:47,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:48,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:49,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:49,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:50,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:51,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:52,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:53,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:54,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:54,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:55,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:55,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:31:55,847 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:31:55,847 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:31:55,849 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:31:55,912 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:31:56,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:57,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:57,914 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:31:58,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:31:59,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:31:59,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:31:59,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:31:59,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:00,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:01,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:02,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:03,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:04,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:04,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:05,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:06,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:07,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:08,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:09,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:09,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:09,921 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:32:10,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:11,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:12,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:13,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:14,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:32:14,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:32:14,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:15,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:15,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:16,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:17,010 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:32:17,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:18,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:19,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:20,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:20,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:21,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:22,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:23,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:24,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:25,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:32:25,418 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:32:25,418 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:32:25,419 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:32:25,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:25,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:25,942 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:32:26,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:27,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:27,943 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:32:28,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:29,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:32:29,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:32:29,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:30,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:31,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:31,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:32,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:33,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:34,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:35,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:36,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:37,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:37,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:38,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:39,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:40,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:41,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:41,952 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:32:42,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:42,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:43,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:44,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:32:44,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:32:44,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:45,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:46,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:47,014 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:32:47,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:48,016 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:48,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:49,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:50,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:51,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:52,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:53,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:53,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:54,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:55,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:56,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:32:56,524 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:32:56,525 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:32:56,526 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:32:56,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:56,971 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:32:57,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:57,971 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:32:58,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:32:58,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:32:59,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:32:59,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:32:59,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:00,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:01,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:02,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:03,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:04,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:04,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:05,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:06,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:07,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:08,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:09,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:10,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:10,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:11,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:12,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:13,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:13,980 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:33:14,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:33:14,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:33:14,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:15,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:15,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:16,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:17,018 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:33:17,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:18,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:19,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:20,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:21,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:21,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:22,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:23,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:24,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:25,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:26,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:26,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:27,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:28,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:33:28,083 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:33:28,084 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:33:28,085 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:33:28,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:29,001 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:33:29,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:33:29,209 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:33:29,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:30,002 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:33:30,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:31,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:32,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:32,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:33,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:34,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:35,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:36,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:37,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:37,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:38,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:39,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:41,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:42,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:43,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:43,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:44,009 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:33:44,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:44,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:33:44,210 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:33:45,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:46,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:47,022 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:33:47,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:48,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:49,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:49,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:50,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:51,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:52,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:53,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:54,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:54,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:33:55,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:56,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:57,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:58,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:58,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:33:58,636 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:33:58,636 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:33:58,638 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:33:59,035 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:33:59,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:33:59,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:33:59,648 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:33:59,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:00,036 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:34:00,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:01,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:02,037 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:34:02,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:03,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:04,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:05,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:05,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:06,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:07,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:08,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:09,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:10,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:11,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:11,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:12,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:13,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:14,046 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:34:14,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:34:14,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:34:14,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:15,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:16,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:16,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:17,026 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:34:17,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:18,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:19,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:20,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:21,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:22,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:22,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:23,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:24,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:25,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:26,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:27,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:27,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:28,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:28,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:34:28,894 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:34:28,894 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:34:28,896 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:34:29,072 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:34:29,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:29,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:34:29,872 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:34:30,074 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:34:30,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:31,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:32,076 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:34:32,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:33,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:33,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:34,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:35,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:36,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:37,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:38,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:38,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:39,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:40,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:41,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:42,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:43,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:44,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:44,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:34:44,862 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:34:44,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:45,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:46,086 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:34:46,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:47,030 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:34:47,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:48,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:49,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:49,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:50,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:51,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:52,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:53,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:54,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:55,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:34:55,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:56,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:57,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:58,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:34:59,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:34:59,867 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:34:59,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:00,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:00,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:01,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:35:01,555 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:35:01,556 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:35:01,557 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:35:01,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:02,103 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:35:02,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:03,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:04,104 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:35:04,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:05,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:06,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:06,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:07,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:08,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:09,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:10,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:11,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:11,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:12,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:13,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:14,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:35:14,867 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:35:14,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:15,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:16,116 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:35:16,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:16,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:17,034 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:35:17,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:18,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:19,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:20,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:21,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:22,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:22,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:23,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:24,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:25,969 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:26,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:27,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:27,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:28,969 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:29,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:35:29,872 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:35:29,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:30,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:31,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:35:31,849 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:35:31,849 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:35:31,850 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:35:32,137 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:35:32,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:33,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:33,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:34,138 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:35:34,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:35,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:36,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:37,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:38,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:38,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:39,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:40,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:41,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:42,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:43,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:43,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:44,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:44,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:35:44,872 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:35:45,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:46,147 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:35:46,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:47,038 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:35:47,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:48,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:49,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:49,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:50,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:51,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:52,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:53,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:54,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:54,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:35:55,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:56,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:57,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:58,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:59,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:35:59,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:35:59,871 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:35:59,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:00,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:01,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:02,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:02,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:36:02,964 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:36:02,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:36:02,965 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:36:03,167 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:36:03,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:04,168 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:36:04,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:05,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:05,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:06,170 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:36:06,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:07,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:08,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:09,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:10,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:10,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:11,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:12,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:13,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:14,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:14,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:36:14,871 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:36:15,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:16,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:16,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:17,042 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:36:17,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:18,180 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:36:18,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:19,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:20,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:21,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:21,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:22,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:23,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:24,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:25,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:26,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:26,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:27,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:28,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:29,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:29,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:36:29,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:36:30,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:31,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:31,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:36:31,763 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:36:31,764 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:36:31,765 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:36:32,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:32,204 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:36:32,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:33,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:34,206 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:36:34,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:35,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:36,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:37,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:37,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:38,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:39,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:40,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:41,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:42,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:43,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:43,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:44,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:44,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:36:44,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:36:45,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:46,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:47,046 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:36:47,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:48,216 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:36:48,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:49,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:49,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:50,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:51,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:52,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:53,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:54,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:54,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:36:55,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:56,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:57,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:58,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:59,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:36:59,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:36:59,879 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:36:59,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:00,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:01,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:02,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:03,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:03,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:37:03,661 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:37:03,662 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:37:03,663 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:37:04,235 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:37:04,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:05,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:05,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:06,237 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:37:06,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:07,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:08,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:09,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:10,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:10,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:11,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:12,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:13,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:14,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:14,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:37:14,879 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:37:15,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:15,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:16,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:17,049 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:37:17,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:18,245 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:37:18,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:19,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:20,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:21,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:21,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:22,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:23,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:24,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:25,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:26,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:27,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:27,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:28,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:29,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:29,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:37:29,881 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:37:30,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:31,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:32,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:33,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:33,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:34,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:34,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:37:34,470 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:37:34,471 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:37:34,473 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:37:35,269 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:37:35,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:36,270 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:37:36,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:37,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:38,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:38,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:39,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:40,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:41,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:42,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:43,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:44,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:44,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:37:44,881 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:37:45,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:45,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:46,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:47,053 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:37:47,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:48,279 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:37:48,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:49,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:50,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:50,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:51,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:52,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:53,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:54,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:55,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:56,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:56,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:37:57,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:58,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:59,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:37:59,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:37:59,892 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:38:00,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:01,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:02,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:02,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:03,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:03,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:38:03,559 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:38:03,560 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:38:03,562 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:38:04,301 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:38:04,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:05,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:06,303 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:38:06,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:07,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:07,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:08,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:09,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:10,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:11,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:12,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:12,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:13,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:14,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:14,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:38:14,882 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:38:15,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:16,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:17,057 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:38:17,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:18,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:18,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:19,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:20,318 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:38:20,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:21,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:22,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:23,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:23,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:24,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:25,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:26,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:27,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:28,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:29,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:29,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:38:29,882 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:38:29,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:30,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:31,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:32,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:33,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:34,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:35,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:35,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:36,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:37,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:38,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:38:38,143 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:38:38,144 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:38:38,146 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:38:38,339 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:38:38,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:39,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:40,340 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:38:40,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:40,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:41,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:42,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:43,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:44,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:44,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:38:44,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:38:45,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:46,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:46,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:47,061 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:38:47,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:48,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:49,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:50,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:51,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:51,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:52,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:53,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:54,350 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:38:54,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:55,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:56,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:56,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:38:57,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:58,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:59,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:38:59,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:38:59,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:39:00,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:01,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:02,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:02,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:03,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:04,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:05,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:06,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:07,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:07,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:08,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:09,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:10,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:39:10,368 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:39:10,368 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:39:10,369 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:39:10,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:11,366 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:39:11,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:12,367 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:39:12,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:13,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:13,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:14,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:14,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:39:14,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:39:15,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:16,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:17,065 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:39:17,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:18,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:19,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:19,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:20,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:21,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:22,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:23,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:24,375 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:39:24,727 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:24,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:25,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:26,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:27,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:28,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:29,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:29,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:39:29,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:39:29,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:30,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:31,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:32,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:33,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:34,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:35,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:35,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:36,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:37,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:38,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:39,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:39,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:39:39,867 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:39:39,867 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:39:39,868 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:39:40,396 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:39:40,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:41,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:41,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:42,398 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:39:42,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:43,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:44,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:44,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:39:44,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:39:45,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:46,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:47,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:47,069 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:39:47,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:48,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:49,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:50,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:51,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:52,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:53,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:53,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:54,406 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:39:54,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:55,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:56,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:57,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:58,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:39:58,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:59,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:39:59,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:39:59,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:40:00,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:01,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:02,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:03,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:03,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:04,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:05,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:06,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:07,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:08,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:09,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:09,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:10,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:11,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:40:11,687 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:40:11,687 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:40:11,688 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:40:12,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:12,428 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:40:12,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:13,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:14,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:14,430 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:40:14,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:14,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:40:14,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:40:15,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:16,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:17,073 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:40:17,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:18,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:19,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:20,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:20,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:21,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:22,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:23,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:24,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:25,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:25,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:26,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:27,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:28,442 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:40:28,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:29,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:29,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:40:29,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:40:30,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:31,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:31,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:32,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:33,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:34,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:35,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:36,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:37,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:37,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:38,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:39,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:40,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:41,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:42,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:42,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:40:42,840 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:40:42,841 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:40:42,843 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:40:43,250 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:43,462 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:40:43,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:44,463 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:40:44,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:44,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:40:44,883 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:40:45,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:46,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:47,077 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:40:47,083 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:47,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:48,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:49,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:50,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:51,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:52,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:53,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:53,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:54,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:55,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:56,471 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:40:56,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:57,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:58,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:40:58,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:59,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:40:59,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:40:59,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:41:00,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:01,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:02,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:03,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:04,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:04,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:05,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:06,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:07,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:08,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:09,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:09,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:10,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:11,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:11,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:41:11,978 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:41:11,978 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:41:11,979 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:41:12,491 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:41:12,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:13,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:14,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:14,493 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:41:14,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:14,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:41:14,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:41:15,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:16,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:17,082 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:41:17,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:18,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:19,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:20,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:20,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:21,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:22,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:23,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:24,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:25,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:26,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:26,501 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:41:26,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:27,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:28,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:29,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:29,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:41:29,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:41:30,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:31,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:31,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:32,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:33,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:34,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:35,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:36,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:37,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:37,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:38,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:39,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:40,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:41,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:41:41,724 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:41:41,724 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:41:41,725 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:41:42,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:42,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:42,522 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:41:43,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:44,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:44,524 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:41:44,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:41:44,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:41:45,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:46,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:47,085 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:41:47,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:48,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:48,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:49,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:50,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:51,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:52,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:53,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:54,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:54,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:55,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:56,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:56,532 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:41:57,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:58,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:59,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:41:59,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:41:59,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:41:59,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:42:00,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:01,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:02,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:03,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:04,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:04,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:05,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:06,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:07,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:08,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:09,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:10,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:10,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:11,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:12,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:13,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:13,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:42:13,470 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:42:13,470 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:42:13,471 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:42:13,552 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:42:14,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:14,553 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:42:14,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:42:14,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:42:15,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:15,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:16,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:16,555 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:42:17,089 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:42:17,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:18,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:19,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:20,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:21,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:21,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:22,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:23,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:24,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:25,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:26,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:27,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:27,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:28,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:29,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:29,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:42:29,884 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:42:30,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:30,564 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:42:31,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:32,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:32,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:33,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:34,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:35,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:36,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:37,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:37,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:38,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:39,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:40,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:41,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:42,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:43,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:43,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:44,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:44,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:42:44,888 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:42:45,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:46,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:47,098 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:42:47,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:48,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:48,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:49,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:49,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:42:49,371 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:42:49,371 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:42:49,373 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:42:49,586 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:42:50,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:50,587 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:42:51,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:52,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:53,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:54,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:42:54,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:55,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:56,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:57,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:58,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:59,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:42:59,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:42:59,886 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:43:00,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:00,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:01,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:02,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:02,595 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:43:03,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:04,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:05,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:06,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:06,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:07,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:08,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:09,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:10,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:11,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:12,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:12,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:13,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:14,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:14,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:43:14,893 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:43:15,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:16,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:17,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:17,102 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:43:17,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:18,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:18,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:43:18,728 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:43:18,728 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:43:18,730 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:43:19,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:19,613 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:43:20,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:20,614 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:43:21,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:22,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:23,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:23,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:24,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:25,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:26,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:27,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:28,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:29,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:29,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:29,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:43:29,889 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:43:30,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:31,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:32,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:32,622 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:43:33,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:34,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:34,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:35,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:36,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:37,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:38,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:39,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:40,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:40,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:41,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:42,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:43,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:44,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:44,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:43:44,891 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:43:45,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:45,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:46,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:47,120 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:43:47,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:48,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:49,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:49,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:43:49,908 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:43:49,909 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:43:49,910 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:43:50,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:50,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:50,645 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:43:51,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:52,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:52,646 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:43:53,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:54,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:55,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:56,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:43:56,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:57,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:58,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:59,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:43:59,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:43:59,891 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:44:00,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:01,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:02,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:02,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:03,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:04,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:04,653 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:44:05,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:06,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:07,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:07,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:08,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:09,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:10,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:11,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:12,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:13,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:13,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:14,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:14,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:44:14,891 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:44:15,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:16,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:17,119 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:44:17,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:18,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:18,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:19,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:20,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:21,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:21,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:44:21,460 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:44:21,461 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:44:21,463 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:44:21,674 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:44:22,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:22,675 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:44:23,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:23,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:24,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:25,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:26,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:27,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:28,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:29,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:29,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:29,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:44:29,891 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:44:30,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:31,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:32,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:33,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:34,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:34,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:34,692 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:44:35,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:36,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:37,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:38,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:39,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:40,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:40,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:41,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:42,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:43,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:44,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:44,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:44:44,907 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:44:45,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:45,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:46,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:47,117 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:44:47,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:48,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:49,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:50,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:50,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:44:50,690 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:44:50,690 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:44:50,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:50,696 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:44:50,711 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:44:51,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:52,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:52,713 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:44:53,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:54,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:55,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:56,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:44:56,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:57,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:58,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:59,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:44:59,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:44:59,897 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:45:00,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:01,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:02,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:02,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:03,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:04,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:05,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:06,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:06,722 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:45:07,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:07,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:08,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:09,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:10,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:11,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:12,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:12,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:13,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:14,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:14,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:45:14,897 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:45:15,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:16,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:17,128 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:45:17,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:17,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:18,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:19,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:20,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:21,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:22,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:22,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:45:22,429 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:45:22,430 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:45:22,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:22,433 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:45:22,742 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:45:23,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:24,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:24,743 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:45:25,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:26,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:27,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:28,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:28,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:29,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:29,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:45:29,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:45:30,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:31,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:32,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:33,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:34,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:34,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:35,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:36,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:36,750 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:45:37,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:38,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:39,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:39,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:40,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:41,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:42,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:43,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:44,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:44,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:45:44,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:45:44,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:45,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:46,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:47,132 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:45:47,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:48,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:49,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:50,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:50,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:51,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:52,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:53,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:53,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:45:53,842 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:45:53,843 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:45:53,847 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:45:54,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:54,771 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:45:54,771 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:45:55,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:45:55,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:56,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:56,772 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:45:57,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:58,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:59,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:45:59,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:45:59,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:46:00,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:00,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:01,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:02,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:03,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:04,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:05,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:05,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:06,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:06,777 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:46:07,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:08,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:09,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:10,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:11,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:11,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:12,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:13,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:14,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:14,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:46:14,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:46:15,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:16,345 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:17,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:17,137 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:46:17,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:18,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:19,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:20,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:21,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:22,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:22,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:23,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:46:23,094 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:46:23,094 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:46:23,096 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:46:23,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:23,798 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:46:24,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:24,799 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:46:25,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:26,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:27,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:28,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:28,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:29,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:29,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:46:29,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:46:30,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:31,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:32,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:33,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:34,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:34,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:35,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:36,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:37,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:38,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:38,808 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:46:39,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:39,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:40,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:41,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:42,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:43,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:44,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:44,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:46:44,899 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:46:44,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:45,513 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:46,514 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:47,140 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:46:47,514 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:48,514 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:49,514 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:50,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:50,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:51,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:52,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:53,520 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:54,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:55,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:46:55,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:56,100 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:46:56,102 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:46:56,103 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:46:56,108 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:46:56,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:56,834 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:46:56,834 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:46:57,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:58,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:58,835 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:46:59,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:46:59,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:46:59,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:47:00,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:01,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:01,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:02,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:03,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:04,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:05,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:06,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:06,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:07,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:08,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:09,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:10,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:10,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:47:11,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:12,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:12,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:13,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:14,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:14,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:47:14,898 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:47:15,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:16,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:17,145 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:47:17,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:17,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:18,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:19,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:20,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:21,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:22,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:23,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:23,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:24,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:25,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:26,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:27,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:47:27,333 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:47:27,333 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:47:27,339 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:47:27,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:27,869 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:47:28,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:28,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:28,870 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:47:29,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:29,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:47:29,899 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:47:30,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:31,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:32,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:33,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:34,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:34,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:35,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:36,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:37,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:38,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:39,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:39,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:40,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:41,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:42,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:42,878 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:47:43,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:44,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:44,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:47:44,899 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:47:45,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:45,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:46,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:47,149 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:47:47,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:48,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:49,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:50,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:50,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:51,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:52,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:53,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:54,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:55,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:47:55,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:56,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:57,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:58,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:47:59,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:47:59,709 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:47:59,709 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:47:59,712 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:47:59,905 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:48:00,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:00,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:00,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:48:00,556 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:48:00,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:48:01,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:02,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:02,908 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:48:03,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:04,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:05,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:06,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:06,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:07,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:08,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:09,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:10,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:11,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:11,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:12,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:13,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:14,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:15,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:48:15,551 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:48:15,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:16,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:16,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:16,916 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:48:17,153 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:48:17,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:18,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:19,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:20,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:21,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:22,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:22,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:23,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:24,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:25,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:26,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:27,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:27,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:28,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:29,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:30,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:48:30,553 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:48:30,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:31,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:32,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:48:32,022 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:48:32,023 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:48:32,028 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:48:32,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:32,935 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:48:32,936 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:48:33,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:33,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:34,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:34,937 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:48:35,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:36,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:37,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:38,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:38,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:39,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:40,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:41,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:42,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:43,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:43,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:44,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:44,947 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:48:45,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:48:45,554 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:48:45,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:46,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:47,156 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:48:47,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:48,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:49,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:49,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:50,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:51,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:52,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:53,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:54,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:54,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:55,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:56,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:57,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:58,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:48:59,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:48:59,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:00,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:49:00,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:49:00,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:01,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:49:01,400 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:49:01,400 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:49:01,402 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:49:01,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:01,968 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:49:02,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:02,968 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:49:03,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:04,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:04,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:05,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:06,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:07,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:08,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:09,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:10,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:10,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:11,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:12,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:13,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:14,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:15,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:49:15,559 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:49:15,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:15,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:16,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:16,977 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:49:17,160 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:49:17,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:18,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:19,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:20,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:21,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:21,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:22,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:23,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:24,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:25,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:26,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:27,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:27,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:28,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:29,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:30,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:49:30,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:49:30,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:31,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:32,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:32,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:33,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:34,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:49:34,124 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:49:34,125 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:49:34,130 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:49:34,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:34,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:49:34,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:49:35,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:36,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:36,997 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:49:37,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:38,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:38,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:39,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:40,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:41,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:42,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:43,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:44,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:44,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:45,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:49:45,563 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:49:45,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:46,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:47,164 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:49:47,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:48,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:49,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:49,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:50,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:51,007 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:49:51,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:52,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:53,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:54,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:55,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:49:55,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:56,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:57,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:58,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:49:59,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:00,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:50:00,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:50:00,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:00,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:01,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:02,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:03,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:04,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:04,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:50:04,956 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:50:04,957 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:50:04,960 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:50:05,033 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:50:05,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:06,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:06,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:07,034 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:50:07,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:08,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:09,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:10,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:11,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:11,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:12,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:13,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:14,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:15,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:50:15,567 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:50:15,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:16,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:16,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:17,168 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:50:17,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:18,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:19,041 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:50:19,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:20,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:21,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:22,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:22,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:23,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:24,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:25,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:26,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:27,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:28,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:28,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:29,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:30,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:50:30,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:50:30,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:31,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:32,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:33,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:33,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:33,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:50:33,936 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:50:33,936 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:50:33,938 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:50:34,063 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:50:34,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:35,064 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:50:35,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:36,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:37,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:38,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:39,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:39,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:40,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:41,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:42,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:43,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:44,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:45,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:45,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:50:45,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:50:45,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:46,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:47,071 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:50:47,172 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:50:47,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:48,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:49,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:50,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:50,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:51,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:52,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:53,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:54,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:55,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:56,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:50:56,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:57,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:58,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:50:59,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:00,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:51:00,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:51:00,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:01,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:01,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:02,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:03,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:03,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:51:03,861 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:51:03,862 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:51:03,864 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:51:04,091 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:51:04,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:05,092 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:51:05,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:06,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:07,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:07,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:08,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:09,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:10,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:11,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:12,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:12,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:13,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:14,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:15,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:51:15,568 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:51:15,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:16,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:17,099 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:51:17,176 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:51:17,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:18,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:18,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:19,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:20,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:21,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:22,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:23,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:23,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:24,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:25,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:26,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:27,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:28,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:28,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:29,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:30,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:51:30,280 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:51:30,281 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:51:30,282 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:51:30,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:51:30,683 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:51:30,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:31,121 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:51:31,121 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:51:31,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:32,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:33,122 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:51:33,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:33,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:34,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:35,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:36,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:37,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:38,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:39,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:39,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:40,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:41,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:42,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:43,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:44,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:44,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:45,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:51:45,678 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:51:45,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:46,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:47,132 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:51:47,180 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:51:47,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:48,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:49,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:50,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:50,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:51,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:52,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:53,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:54,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:55,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:51:55,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:56,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:57,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:58,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:51:59,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:00,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:00,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:52:00,682 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:52:00,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:01,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:02,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:03,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:52:03,709 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:52:03,709 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:52:03,710 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:52:04,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:04,151 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:52:05,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:05,153 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:52:05,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:06,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:07,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:08,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:09,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:10,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:11,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:11,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:12,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:13,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:14,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:15,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:15,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:52:15,682 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:52:16,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:16,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:17,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:17,184 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:52:18,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:19,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:19,161 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:52:20,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:21,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:22,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:22,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:23,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:24,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:25,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:26,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:27,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:28,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:28,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:29,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:30,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:30,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:52:30,699 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:52:31,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:32,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:33,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:33,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:34,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:35,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:36,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:36,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:52:36,195 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:52:36,195 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:52:36,196 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:52:37,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:37,193 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:52:37,193 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:52:38,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:39,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:39,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:40,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:41,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:42,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:43,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:44,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:44,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:45,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:45,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:52:45,687 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:52:46,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:47,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:47,188 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:52:48,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:49,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:49,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:50,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:51,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:51,204 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:52:52,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:53,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:54,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:55,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:55,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:52:56,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:57,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:58,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:52:59,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:00,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:00,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:00,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:53:00,700 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:53:01,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:02,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:03,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:04,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:05,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:06,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:06,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:53:06,247 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:53:06,247 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:53:06,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:06,248 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:53:07,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:07,223 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:53:07,224 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:53:08,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:09,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:10,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:11,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:11,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:12,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:13,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:14,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:15,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:15,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:53:15,690 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:53:16,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:16,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:17,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:17,192 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:53:18,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:19,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:20,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:21,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:21,233 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:53:22,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:22,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:23,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:24,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:25,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:26,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:27,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:28,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:28,691 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:29,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:30,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:30,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:53:30,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:53:31,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:32,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:33,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:33,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:34,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:35,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:36,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:53:36,123 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:53:36,123 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:53:36,125 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:53:36,252 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:53:36,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:37,254 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:53:37,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:38,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:39,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:39,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:40,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:41,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:42,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:43,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:44,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:44,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:45,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:45,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:53:45,694 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:53:46,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:47,196 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:53:47,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:48,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:49,261 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:53:49,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:50,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:50,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:51,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:52,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:53,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:54,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:55,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:55,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:53:56,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:57,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:58,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:53:59,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:00,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:00,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:54:00,699 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:54:00,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:01,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:02,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:03,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:04,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:05,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:06,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:06,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:54:06,298 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:54:06,299 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:54:06,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:06,300 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:54:07,280 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:54:07,281 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:54:07,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:08,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:09,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:10,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:11,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:11,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:12,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:13,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:14,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:15,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:15,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:54:15,697 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:54:16,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:16,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:17,200 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:54:17,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:18,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:19,284 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:19,289 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:54:20,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:21,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:22,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:22,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:23,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:24,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:25,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:26,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:27,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:28,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:28,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:29,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:30,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:30,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:54:30,704 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:54:31,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:32,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:33,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:33,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:34,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:35,294 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:36,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:54:36,117 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:54:36,118 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:54:36,119 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:54:36,312 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:54:36,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:37,312 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:54:37,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:38,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:39,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:39,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:40,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:41,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:42,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:43,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:44,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:44,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:45,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:45,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:54:45,701 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:54:46,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:47,204 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:54:47,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:48,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:49,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:50,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:50,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:51,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:52,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:53,322 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:54:53,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:54,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:55,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:55,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:54:56,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:57,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:58,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:54:59,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:00,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:00,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:55:00,701 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:55:00,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:01,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:02,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:03,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:04,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:05,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:06,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:06,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:07,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:08,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:08,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:55:08,874 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:55:08,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:55:08,876 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:55:09,342 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:55:09,343 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:55:09,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:10,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:11,344 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:55:11,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:11,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:12,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:13,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:14,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:15,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:15,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:55:15,701 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:55:16,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:16,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:17,208 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:55:17,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:18,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:19,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:20,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:21,350 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:55:21,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:22,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:22,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:23,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:24,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:25,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:26,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:27,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:28,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:28,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:29,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:30,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:55:30,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:55:30,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:31,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:32,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:33,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:33,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:34,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:35,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:36,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:37,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:38,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:55:38,030 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:55:38,031 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:55:38,032 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:55:38,365 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:55:38,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:39,367 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:55:39,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:39,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:40,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:41,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:42,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:43,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:44,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:44,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:45,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:55:45,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:55:45,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:46,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:47,212 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:55:47,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:48,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:49,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:49,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:50,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:51,374 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:55:51,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:52,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:53,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:54,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:55,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:55:55,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:56,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:57,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:58,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:55:59,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:00,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:56:00,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:56:00,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:00,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:01,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:02,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:03,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:04,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:05,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:06,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:06,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:07,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:56:07,087 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:56:07,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:56:07,088 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:56:07,396 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:56:07,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:08,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:09,398 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:56:09,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:10,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:11,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:12,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:12,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:13,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:14,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:15,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:56:15,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:56:15,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:16,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:17,216 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:56:17,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:18,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:18,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:19,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:20,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:21,406 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:56:21,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:22,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:23,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:23,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:24,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:25,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:26,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:27,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:28,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:29,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:29,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:30,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:56:30,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:56:30,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:31,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:32,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:33,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:34,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:34,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:35,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:36,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:37,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:56:37,338 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:56:37,339 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:56:37,340 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:56:37,429 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:56:37,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:38,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:39,431 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:56:39,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:40,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:40,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:41,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:42,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:43,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:44,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:45,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:56:45,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:56:45,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:45,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:46,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:47,220 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:56:47,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:48,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:49,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:50,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:51,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:51,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:52,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:53,440 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:56:53,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:54,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:55,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:56,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:56:56,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:57,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:58,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:56:59,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:00,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:57:00,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:57:00,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:01,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:01,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:02,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:03,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:04,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:05,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:06,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:07,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:07,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:07,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:57:07,939 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:57:07,940 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:57:07,941 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:57:08,468 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:57:08,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:09,468 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:57:09,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:10,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:11,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:12,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:13,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:13,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:14,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:15,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:57:15,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:57:15,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:16,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:17,224 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:57:17,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:18,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:19,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:19,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:20,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:21,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:22,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:23,477 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:57:23,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:24,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:24,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:25,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:26,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:27,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:28,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:29,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:29,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:30,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:57:30,702 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:57:30,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:31,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:32,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:33,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:34,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:34,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:35,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:36,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:37,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:38,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:39,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:57:39,859 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:57:39,859 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:57:39,860 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:57:39,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:40,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:40,496 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:57:41,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:41,498 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:57:42,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:43,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:44,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:45,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:45,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:57:45,703 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:57:45,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:46,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:47,228 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:57:47,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:48,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:49,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:50,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:51,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:51,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:52,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:53,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:53,505 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:57:54,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:55,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:56,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:56,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:57:57,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:58,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:57:59,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:00,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:00,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:58:00,703 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:58:01,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:01,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:02,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:03,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:04,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:05,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:06,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:07,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:07,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:08,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:09,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:10,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:58:10,100 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:58:10,101 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:58:10,102 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:58:10,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:10,522 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:58:11,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:11,523 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:58:12,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:13,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:13,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:14,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:15,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:15,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:58:15,703 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:58:16,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:17,232 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:58:17,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:18,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:19,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:19,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:20,509 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:21,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:22,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:23,532 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:58:23,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:24,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:24,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:25,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:26,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:27,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:28,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:29,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:29,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:30,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:30,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:58:30,703 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:58:31,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:32,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:33,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:34,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:34,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:35,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:36,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:37,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:38,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:38,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:58:38,978 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:58:38,978 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:58:38,980 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:58:39,554 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:58:39,555 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:58:39,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:40,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:40,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:41,556 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:58:41,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:42,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:43,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:44,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:45,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:45,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:58:45,703 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:58:45,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:46,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:47,235 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:58:47,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:48,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:49,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:50,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:51,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:51,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:52,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:53,566 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:58:53,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:54,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:55,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:56,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:57,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:57,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:58:58,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:58:59,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:00,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:00,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:59:00,703 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:59:01,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:02,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:02,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:03,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:04,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:05,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:06,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:07,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:08,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:08,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:09,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:10,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:59:10,188 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:59:10,189 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:59:10,190 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:59:10,588 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:59:10,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:11,589 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:59:11,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:12,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:13,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:14,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:14,779 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:15,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:15,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:59:15,703 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:59:16,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:17,240 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:59:17,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:18,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:19,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:20,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:20,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:21,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:22,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:23,597 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:59:23,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:24,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:25,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:25,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:26,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:27,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:28,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:29,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:30,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:30,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:59:30,704 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:59:30,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:31,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:32,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:33,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:34,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:35,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:36,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:36,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:37,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:38,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:39,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:40,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:40,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 12:59:40,729 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 12:59:40,729 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 12:59:40,731 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 12:59:41,619 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:59:41,619 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 12:59:41,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:42,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:42,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:43,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:44,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:45,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:45,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 12:59:45,704 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 12:59:46,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:47,243 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 12:59:47,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:47,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:48,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:49,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:50,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:51,626 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 12:59:51,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:52,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:52,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:53,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:54,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:55,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:56,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:57,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:58,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 12:59:58,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 12:59:59,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:00,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:00,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:00:00,706 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:00:01,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:02,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:03,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:03,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:04,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:04,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:00:04,943 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:00:04,944 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:00:04,945 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:00:05,645 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:00:05,646 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:00:05,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:06,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:07,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:08,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:09,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:09,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:10,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:11,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:12,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:13,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:14,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:15,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:15,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:00:15,706 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:00:15,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:16,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:17,247 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:00:17,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:18,673 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:19,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:20,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:20,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:21,656 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:00:21,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:22,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:23,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:24,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:25,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:25,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:26,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:27,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:28,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:29,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:30,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:30,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:00:30,706 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:00:30,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:31,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:32,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:33,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:34,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:35,681 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:35,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:36,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:37,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:38,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:38,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:00:38,955 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:00:38,955 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:00:38,956 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:00:39,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:39,682 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:00:39,683 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:00:40,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:41,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:41,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:42,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:43,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:44,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:45,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:45,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:00:45,706 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:00:46,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:47,251 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:00:47,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:47,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:48,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:49,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:50,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:51,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:52,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:52,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:53,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:53,691 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:00:54,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:55,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:56,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:57,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:57,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:00:58,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:00:59,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:00,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:00,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:01:00,706 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:01:01,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:02,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:02,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:03,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:04,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:05,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:06,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:07,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:07,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:01:07,725 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:01:07,726 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:01:07,728 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:01:08,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:08,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:08,713 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:01:09,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:09,714 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:01:10,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:11,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:12,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:13,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:13,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:14,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:15,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:15,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:01:15,707 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:01:16,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:17,255 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:01:17,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:18,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:19,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:19,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:20,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:21,721 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:01:21,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:22,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:23,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:24,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:24,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:25,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:26,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:27,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:28,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:29,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:30,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:01:30,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:01:30,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:30,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:31,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:32,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:33,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:34,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:35,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:35,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:36,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:01:36,095 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:01:36,096 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:01:36,097 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:01:36,746 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:01:36,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:37,747 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:01:37,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:38,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:39,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:40,829 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:40,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:41,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:42,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:43,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:44,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:45,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:01:45,710 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:01:45,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:46,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:46,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:47,260 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:01:47,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:48,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:49,760 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:01:49,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:50,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:51,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:51,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:52,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:53,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:54,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:55,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:56,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:57,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:57,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:01:58,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:01:59,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:00,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:02:00,720 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:02:00,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:01,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:02,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:02,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:03,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:04,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:02:04,491 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:02:04,492 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:02:04,493 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:02:04,806 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:02:04,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:05,807 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:02:05,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:06,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:07,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:07,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:08,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:09,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:10,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:11,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:12,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:12,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:13,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:14,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:15,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:02:15,716 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:02:15,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:16,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:17,263 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:02:17,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:18,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:18,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:19,818 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:02:19,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:20,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:21,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:22,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:23,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:23,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:24,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:25,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:26,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:27,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:28,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:29,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:29,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:30,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:02:30,716 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:02:30,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:31,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:32,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:33,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:34,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:34,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:35,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:36,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:37,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:02:37,105 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:02:37,105 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:02:37,106 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:02:37,839 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:02:37,839 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:02:37,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:38,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:39,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:40,514 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:40,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:41,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:42,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:43,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:44,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:45,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:02:45,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:02:45,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:45,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:46,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:47,267 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:02:47,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:48,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:49,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:50,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:50,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:51,850 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:02:51,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:52,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:53,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:54,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:55,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:56,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:02:56,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:57,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:58,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:02:59,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:00,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:03:00,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:03:00,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:01,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:02,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:02,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:03,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:04,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:05,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:06,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:07,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:07,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:08,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:03:08,025 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:03:08,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:03:08,027 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:03:08,871 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:03:08,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:09,872 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:03:09,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:10,919 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:11,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:12,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:13,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:13,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:14,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:15,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:03:15,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:03:15,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:16,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:17,271 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:03:17,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:18,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:19,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:19,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:20,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:21,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:03:21,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:22,920 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:23,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:24,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:24,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:25,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:26,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:27,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:28,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:29,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:30,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:03:30,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:03:30,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:30,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:31,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:32,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:33,926 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:34,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:35,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:35,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:36,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:37,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:37,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:03:38,001 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:03:38,001 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:03:38,002 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:03:38,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:03:38,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:39,907 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:03:39,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:40,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:40,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:41,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:42,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:43,929 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:44,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:45,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:03:45,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:03:45,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:46,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:46,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:47,275 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:03:47,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:48,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:49,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:50,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:51,914 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:03:51,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:51,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:52,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:53,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:54,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:55,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:56,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:57,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:03:57,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:58,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:03:59,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:00,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:04:00,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:04:00,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:01,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:02,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:02,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:03,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:04,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:05,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:06,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:07,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:07,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:04:07,994 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:04:07,995 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:04:07,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:07,997 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:04:08,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:08,939 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:04:09,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:09,940 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:04:10,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:11,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:12,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:13,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:13,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:14,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:15,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:04:15,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:04:15,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:16,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:17,280 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:04:17,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:18,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:19,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:19,939 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:20,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:21,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:22,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:23,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:24,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:24,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:25,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:26,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:27,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:28,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:29,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:29,940 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:30,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:04:30,717 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:04:30,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:32,955 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:04:33,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:35,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:40,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:40,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:41,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:45,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:47,284 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:04:48,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:04:48,854 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:04:49,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:51,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:56,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:04:56,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:04:58,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:01,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:05,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:05,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:05:05,950 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:05:06,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:06,421 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:11,422 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:13,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:15,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:16,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:17,287 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:05:22,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:27,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:27,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:05:27,033 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:05:27,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:27,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:28,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:28,286 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:05:29,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:30,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:30,290 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:05:31,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:32,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:33,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:33,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:34,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:35,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:36,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:37,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:37,708 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:05:37,709 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:05:38,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:38,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:39,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:40,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:41,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:42,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:43,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:44,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:44,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:45,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:46,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:47,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:47,292 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:05:48,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:49,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:50,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:50,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:51,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:52,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:52,709 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:05:52,709 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:05:53,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:54,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:55,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:55,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:05:56,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:57,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:58,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:05:59,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:00,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:01,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:01,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:02,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:03,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:04,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:05,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:06,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:07,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:07,709 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:06:07,710 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:06:07,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:08,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:09,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:10,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:11,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:12,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:13,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:13,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:14,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:15,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:16,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:17,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:17,296 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:06:18,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:19,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:19,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:20,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:24,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:29,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:34,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:39,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:40,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:06:40,584 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:06:40,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:41,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:42,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:43,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:44,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:44,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:45,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:46,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:47,301 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:06:47,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:48,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:49,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:50,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:50,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:51,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:52,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:53,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:54,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:55,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:06:55,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:06:55,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:55,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:06:56,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:57,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:58,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:06:59,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:00,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:01,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:01,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:02,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:03,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:04,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:05,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:06,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:07,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:07,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:08,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:09,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:10,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:07:10,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:07:10,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:11,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:12,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:12,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:13,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:14,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:15,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:16,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:17,304 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:07:17,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:18,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:18,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:19,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:20,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:21,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:22,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:23,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:23,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:24,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:25,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:07:25,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:07:25,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:26,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:27,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:28,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:28,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:29,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:30,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:31,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:32,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:33,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:34,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:34,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:35,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:36,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:37,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:38,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:39,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:40,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:07:40,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:07:40,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:40,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:41,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:42,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:43,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:44,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:45,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:46,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:46,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:47,308 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:07:47,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:48,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:49,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:50,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:51,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:52,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:52,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:53,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:54,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:55,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:07:55,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:07:55,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:56,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:57,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:57,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:07:58,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:07:59,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:00,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:01,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:02,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:03,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:03,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:04,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:05,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:06,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:07,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:08,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:08,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:09,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:10,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:08:10,579 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:08:10,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:11,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:12,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:13,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:13,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:14,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:15,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:16,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:17,312 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:08:17,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:18,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:19,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:19,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:20,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:21,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:22,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:23,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:24,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:24,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:08:29,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:34,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:39,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:44,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:47,316 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:08:50,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:08:55,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:00,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:03,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:03,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:09:03,203 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:09:04,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:05,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:05,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:06,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:06,292 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:09:07,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:08,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:09,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:10,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:10,295 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:09:10,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:11,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:12,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:13,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:14,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:15,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:16,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:16,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:17,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:17,320 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:09:18,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:09:18,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:18,202 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:09:19,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:20,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:20,302 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:09:21,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:21,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:22,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:23,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:24,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:25,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:26,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:27,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:27,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:28,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:29,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:30,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:31,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:32,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:32,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:33,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:09:33,200 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:09:33,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:34,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:35,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:36,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:36,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:09:36,418 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:09:36,419 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:09:36,420 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:09:37,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:37,317 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:09:37,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:38,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:38,319 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:09:39,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:40,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:41,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:42,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:43,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:43,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:44,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:45,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:46,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:47,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:47,324 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:09:48,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:09:48,200 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:09:48,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:48,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:49,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:50,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:50,335 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:09:51,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:52,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:53,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:54,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:09:54,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:55,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:56,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:57,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:58,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:09:59,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:00,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:00,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:01,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:02,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:03,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:10:03,205 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:10:03,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:04,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:05,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:05,283 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:06,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:06,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:10:06,891 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:10:06,891 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:10:06,892 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:10:07,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:07,360 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:10:08,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:08,361 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:10:09,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:10,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:11,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:11,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:12,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:13,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:14,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:15,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:16,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:16,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:17,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:17,327 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:10:18,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:10:18,205 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:10:18,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:19,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:20,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:21,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:21,370 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:10:21,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:22,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:23,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:24,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:25,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:26,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:27,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:27,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:28,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:29,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:30,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:31,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:32,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:32,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:33,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:10:33,215 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:10:33,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:34,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:35,303 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:35,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:10:35,574 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:10:35,574 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:10:35,576 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:10:36,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:36,391 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:10:37,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:37,392 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:10:37,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:38,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:39,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:40,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:41,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:42,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:43,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:43,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:44,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:45,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:46,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:47,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:47,331 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:10:48,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:10:48,211 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:10:48,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:48,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:49,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:50,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:51,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:51,402 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:10:52,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:53,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:54,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:10:54,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:55,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:56,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:57,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:58,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:10:59,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:00,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:00,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:01,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:02,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:03,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:11:03,214 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:11:03,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:04,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:05,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:05,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:06,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:06,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:11:06,387 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:11:06,387 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:11:06,388 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:11:06,428 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:11:07,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:08,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:09,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:09,430 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:11:10,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:11,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:11,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:12,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:13,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:14,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:15,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:16,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:17,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:17,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:17,335 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:11:18,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:11:18,214 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:11:18,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:19,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:20,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:21,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:22,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:23,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:23,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:23,439 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:11:24,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:25,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:26,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:27,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:28,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:28,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:29,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:30,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:31,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:32,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:33,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:33,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:11:33,237 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:11:33,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:34,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:35,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:36,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:37,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:38,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:38,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:11:38,950 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:11:38,951 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:11:38,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:38,953 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:11:39,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:39,462 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:11:40,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:41,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:41,463 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:11:42,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:43,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:44,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:44,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:45,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:46,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:47,339 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:11:47,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:48,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:11:48,216 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:11:48,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:49,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:49,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:50,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:51,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:52,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:53,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:53,472 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:11:54,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:55,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:11:55,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:56,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:57,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:58,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:11:59,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:00,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:01,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:01,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:02,358 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:03,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:12:03,228 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:12:03,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:04,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:05,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:06,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:06,369 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:07,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:08,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:08,403 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:12:08,405 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:12:08,406 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:12:08,407 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:12:08,492 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:12:09,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:10,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:11,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:11,495 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:12:11,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:12,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:13,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:14,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:15,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:16,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:17,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:17,344 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:12:17,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:18,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:12:18,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:12:18,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:19,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:20,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:21,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:22,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:22,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:23,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:24,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:25,504 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:12:25,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:26,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:27,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:28,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:28,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:29,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:30,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:31,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:32,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:33,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:12:33,223 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:12:33,313 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:33,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:34,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:35,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:36,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:37,549 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:38,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:39,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:39,551 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:40,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:12:40,149 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:12:40,150 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:12:40,151 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:12:40,521 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:12:40,562 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:41,522 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:12:41,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:42,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:43,524 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:12:43,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:44,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:45,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:45,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:46,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:47,347 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:12:47,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:48,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:12:48,223 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:12:48,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:49,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:50,381 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:50,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:51,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:52,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:53,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:54,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:55,531 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:12:55,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:56,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:12:56,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:57,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:58,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:12:59,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:00,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:01,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:02,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:02,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:03,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:13:03,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:13:03,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:04,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:05,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:06,604 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:07,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:07,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:08,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:09,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:10,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:10,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:13:10,742 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:13:10,743 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:13:10,744 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:13:11,550 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:13:11,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:12,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:13,157 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:13,552 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:13:13,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:14,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:15,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:16,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:17,351 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:13:17,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:18,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:13:18,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:13:18,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:18,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:19,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:20,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:21,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:22,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:23,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:24,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:24,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:25,560 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:13:25,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:26,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:27,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:28,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:29,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:30,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:30,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:31,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:32,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:33,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:13:33,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:13:33,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:34,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:35,356 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:35,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:36,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:37,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:38,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:39,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:40,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:41,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:41,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:42,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:13:42,130 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:13:42,131 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:13:42,132 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:13:42,588 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:13:42,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:43,589 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:13:43,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:44,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:45,590 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:13:45,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:46,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:47,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:47,355 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:13:47,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:48,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:13:48,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:13:48,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:49,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:50,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:51,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:52,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:52,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:53,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:54,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:55,596 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:13:55,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:56,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:57,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:57,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:13:58,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:13:59,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:00,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:01,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:02,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:03,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:14:03,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:14:03,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:03,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:04,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:05,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:06,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:07,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:08,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:09,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:09,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:14:09,653 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:14:09,653 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:14:09,655 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:14:10,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:10,615 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:14:11,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:11,615 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:14:12,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:13,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:14,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:15,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:15,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:16,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:17,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:17,359 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:14:18,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:18,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:14:18,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:14:19,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:20,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:20,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:21,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:22,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:23,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:24,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:25,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:25,624 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:14:26,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:26,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:27,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:28,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:29,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:30,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:31,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:32,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:32,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:33,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:14:33,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:14:33,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:34,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:35,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:36,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:37,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:37,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:38,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:39,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:40,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:41,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:41,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:14:41,662 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:14:41,663 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:14:41,664 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:14:42,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:42,645 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:14:43,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:43,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:43,646 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:14:44,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:45,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:46,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:47,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:47,363 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:14:48,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:14:48,224 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:14:48,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:48,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:49,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:50,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:51,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:52,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:53,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:53,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:54,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:55,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:55,654 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:14:56,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:57,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:58,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:14:58,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:14:59,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:00,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:01,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:02,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:03,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:15:03,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:15:03,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:03,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:04,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:05,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:06,323 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:07,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:08,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:09,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:09,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:10,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:10,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:15:10,876 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:15:10,876 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:15:10,877 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:15:11,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:11,674 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:15:12,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:13,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:13,675 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:15:14,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:15,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:15,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:16,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:17,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:17,367 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:15:18,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:15:18,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:15:18,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:19,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:20,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:20,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:21,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:22,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:23,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:24,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:25,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:26,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:26,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:27,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:27,683 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:15:28,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:29,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:30,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:31,320 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:31,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:32,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:33,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:15:33,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:15:33,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:34,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:35,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:36,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:36,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:37,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:38,335 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:39,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:40,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:41,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:42,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:42,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:43,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:43,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:15:43,577 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:15:43,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:15:43,578 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:15:43,711 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:15:44,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:45,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:45,713 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:15:46,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:47,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:47,371 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:15:47,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:48,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:15:48,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:15:48,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:49,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:50,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:51,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:52,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:53,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:53,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:54,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:55,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:56,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:57,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:58,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:15:58,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:59,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:15:59,728 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:16:00,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:01,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:02,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:03,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:16:03,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:16:03,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:04,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:04,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:05,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:06,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:07,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:08,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:09,333 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:09,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:10,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:11,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:12,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:13,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:14,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:14,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:14,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:16:14,857 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:16:14,857 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:16:14,858 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:16:15,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:15,749 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:16:16,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:17,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:17,375 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:16:17,751 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:16:18,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:16:18,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:16:18,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:19,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:20,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:20,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:21,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:22,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:23,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:24,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:25,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:25,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:26,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:27,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:27,757 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:16:28,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:29,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:30,330 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:30,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:31,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:32,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:33,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:16:33,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:16:33,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:34,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:35,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:35,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:36,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:37,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:38,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:39,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:40,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:40,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:41,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:42,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:42,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:16:42,641 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:16:42,641 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:16:42,643 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:16:42,783 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:16:43,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:43,784 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:16:44,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:45,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:45,785 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:16:46,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:46,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:47,379 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:16:47,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:48,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:16:48,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:16:48,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:49,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:50,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:51,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:51,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:52,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:53,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:54,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:55,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:56,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:57,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:16:57,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:57,793 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:16:58,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:16:59,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:00,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:01,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:02,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:02,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:03,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:17:03,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:17:03,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:04,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:05,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:06,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:07,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:08,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:08,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:09,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:10,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:11,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:12,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:13,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:14,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:14,430 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:17:14,432 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:17:14,433 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:17:14,434 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:17:14,814 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:17:14,833 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:15,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:15,814 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:17:16,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:17,384 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:17:17,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:17,816 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:17:18,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:17:18,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:17:18,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:19,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:19,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:20,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:21,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:22,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:23,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:24,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:25,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:25,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:26,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:27,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:27,823 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:17:28,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:29,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:30,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:30,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:31,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:32,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:33,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:17:33,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:17:33,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:34,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:35,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:35,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:36,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:37,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:38,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:39,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:40,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:40,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:41,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:42,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:43,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:43,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:17:43,777 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:17:43,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:17:43,779 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:17:43,846 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:17:44,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:45,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:45,848 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:17:46,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:46,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:47,388 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:17:47,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:48,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:17:48,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:17:48,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:49,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:50,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:51,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:51,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:52,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:53,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:54,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:55,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:56,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:57,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:17:57,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:58,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:17:59,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:00,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:01,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:01,858 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:18:02,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:03,008 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:03,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:18:03,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:18:03,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:04,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:05,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:06,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:07,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:08,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:08,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:09,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:10,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:11,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:12,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:13,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:13,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:14,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:14,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:18:14,755 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:18:14,756 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:18:14,757 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:18:14,876 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:18:15,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:15,877 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:18:16,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:17,392 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:18:17,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:17,878 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:18:18,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:18:18,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:18:18,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:19,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:19,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:20,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:21,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:22,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:23,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:24,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:25,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:25,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:26,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:27,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:27,885 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:18:28,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:29,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:30,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:30,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:31,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:32,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:33,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:18:33,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:18:33,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:34,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:35,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:36,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:36,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:37,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:38,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:39,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:40,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:41,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:41,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:42,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:43,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:44,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:18:44,194 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:18:44,195 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:18:44,196 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:18:44,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:44,907 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:18:45,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:45,908 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:18:46,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:46,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:47,396 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:18:47,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:48,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:18:48,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:18:48,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:49,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:50,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:51,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:52,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:52,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:53,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:54,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:55,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:56,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:57,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:18:57,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:58,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:59,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:18:59,919 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:19:00,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:01,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:02,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:02,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:03,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:19:03,226 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:19:03,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:04,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:05,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:06,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:07,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:08,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:08,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:09,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:10,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:11,701 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:12,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:13,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:13,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:14,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:15,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:16,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:16,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:19:16,993 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:19:16,994 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:19:16,995 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:19:17,400 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:19:17,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:17,947 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:19:18,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:19:18,227 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:19:18,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:18,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:19,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:19,948 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:19:20,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:21,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:22,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:23,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:24,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:24,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:25,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:26,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:27,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:28,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:29,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:30,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:30,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:31,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:31,955 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:19:32,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:33,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:19:33,227 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:19:33,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:34,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:35,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:35,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:36,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:37,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:38,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:39,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:40,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:41,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:41,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:42,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:43,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:44,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:45,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:46,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:46,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:47,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:19:47,081 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:19:47,081 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:19:47,083 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:19:47,403 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:19:47,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:47,971 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:19:48,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:19:48,227 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:19:48,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:49,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:49,973 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:19:50,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:51,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:51,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:52,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:53,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:54,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:55,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:56,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:57,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:19:57,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:58,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:19:59,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:00,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:01,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:01,980 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:20:02,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:03,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:20:03,227 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:20:03,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:03,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:04,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:05,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:06,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:07,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:08,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:08,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:09,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:10,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:11,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:12,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:13,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:13,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:14,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:15,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:16,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:17,406 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:20:17,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:18,007 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:20:18,009 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:20:18,009 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:20:18,011 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:20:18,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:20:18,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:20:18,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:18,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:18,997 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:20:19,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:19,997 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:20:20,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:21,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:22,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:23,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:24,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:24,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:25,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:26,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:27,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:28,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:29,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:30,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:30,719 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:31,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:32,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:33,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:20:33,413 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:20:33,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:34,007 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:20:34,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:35,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:35,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:36,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:37,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:38,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:39,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:40,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:41,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:41,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:42,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:43,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:44,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:45,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:46,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:46,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:47,407 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:20:47,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:48,419 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:20:48,420 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:20:48,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:49,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:49,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:20:49,932 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:20:49,933 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:20:49,938 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:20:50,022 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:20:50,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:51,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:52,024 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:20:52,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:52,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:53,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:54,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:55,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:56,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:57,600 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:20:57,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:58,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:20:59,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:00,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:01,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:02,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:03,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:03,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:21:03,417 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:21:03,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:04,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:05,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:06,032 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:21:06,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:07,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:08,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:08,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:09,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:10,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:11,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:12,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:13,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:14,490 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:14,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:15,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:16,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:17,408 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:21:17,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:18,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:21:18,426 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:21:18,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:19,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:19,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:20,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:21,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:21:21,224 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:21:21,225 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:21:21,231 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:21:21,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:22,048 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:21:22,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:23,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:24,049 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:21:24,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:25,492 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:25,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:26,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:27,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:28,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:29,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:30,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:31,492 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:31,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:32,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:33,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:21:33,423 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:21:33,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:34,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:35,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:36,058 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:21:36,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:36,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:37,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:38,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:39,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:40,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:41,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:42,709 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:42,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:43,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:44,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:45,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:46,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:47,419 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:21:47,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:48,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:48,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:21:48,425 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:21:48,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:49,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:50,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:51,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:21:51,149 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:21:51,149 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:21:51,150 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:21:51,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:52,076 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:21:52,076 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:21:52,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:53,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:53,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:54,077 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:21:54,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:55,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:56,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:57,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:58,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:21:58,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:21:59,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:00,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:01,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:02,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:03,425 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:22:03,425 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:22:03,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:04,084 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:22:04,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:04,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:05,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:06,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:07,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:08,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:09,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:09,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:10,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:11,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:12,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:13,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:14,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:14,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:15,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:16,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:17,422 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:22:17,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:18,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:22:18,430 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:22:18,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:19,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:20,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:22:20,111 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:22:20,112 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:22:20,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:20,114 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:22:20,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:21,100 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:22:21,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:22,101 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:22:22,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:23,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:24,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:25,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:25,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:26,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:27,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:28,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:29,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:30,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:30,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:31,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:32,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:33,426 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:22:33,426 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:22:33,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:34,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:35,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:36,109 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:22:36,406 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:36,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:37,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:38,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:39,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:40,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:41,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:41,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:42,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:43,894 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:44,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:45,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:46,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:46,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:47,426 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:22:47,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:48,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:22:48,432 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:22:48,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:49,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:50,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:22:50,369 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:22:50,370 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:22:50,371 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:22:50,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:51,126 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:22:51,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:51,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:52,127 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:22:52,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:53,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:54,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:55,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:56,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:57,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:22:57,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:58,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:22:59,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:00,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:01,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:02,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:03,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:03,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:23:03,432 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:23:03,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:04,135 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:23:04,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:05,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:06,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:07,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:08,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:08,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:09,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:10,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:11,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:12,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:13,718 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:13,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:14,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:15,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:16,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:17,425 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:23:17,914 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:18,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:23:18,441 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:23:18,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:18,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:19,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:20,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:21,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:23:21,509 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:23:21,509 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:23:21,511 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:23:21,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:22,156 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:23:22,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:23,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:24,158 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:23:24,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:24,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:25,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:26,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:27,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:28,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:29,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:30,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:30,922 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:31,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:32,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:33,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:23:33,436 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:23:33,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:34,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:35,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:36,166 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:23:36,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:36,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:37,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:38,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:39,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:40,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:41,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:41,923 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:42,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:43,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:44,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:45,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:46,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:46,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:47,427 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:23:47,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:48,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:23:48,436 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:23:48,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:49,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:50,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:51,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:51,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:52,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:23:52,470 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:23:52,470 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:23:52,472 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:23:52,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:53,182 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:23:53,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:54,183 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:23:54,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:55,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:56,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:57,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:23:57,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:58,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:23:59,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:00,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:01,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:02,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:02,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:03,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:24:03,436 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:24:04,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:04,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:05,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:06,190 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:24:06,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:07,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:08,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:08,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:09,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:10,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:11,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:12,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:13,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:13,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:14,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:15,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:16,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:17,438 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:24:18,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:18,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:24:18,439 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:24:18,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:19,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:24:19,373 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:24:19,374 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:24:19,374 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:19,375 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:24:19,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:20,213 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:24:20,213 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:24:20,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:21,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:22,214 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:24:22,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:23,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:24,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:24,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:25,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:26,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:27,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:28,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:29,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:29,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:30,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:31,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:32,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:33,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:24:33,440 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:24:33,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:34,221 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:24:34,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:35,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:35,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:36,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:37,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:38,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:39,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:40,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:40,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:41,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:42,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:44,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:44,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:45,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:45,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:47,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:47,436 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:24:48,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:48,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:24:48,444 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:24:49,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:49,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:24:49,804 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:24:49,804 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:24:49,806 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:24:50,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:50,239 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:24:51,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:51,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:52,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:52,241 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:24:53,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:54,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:55,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:56,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:56,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:24:57,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:58,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:24:59,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:00,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:01,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:02,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:02,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:03,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:03,444 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:25:03,444 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:25:04,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:05,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:06,250 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:25:06,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:07,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:08,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:08,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:09,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:10,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:11,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:12,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:13,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:13,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:14,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:15,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:16,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:17,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:17,445 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:25:18,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:18,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:25:18,448 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:25:18,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:19,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:20,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:21,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:22,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:23,123 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:25:23,124 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:25:23,124 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:25:23,126 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:25:23,270 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:25:23,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:23,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:24,271 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:25:24,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:25,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:26,272 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:25:26,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:27,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:28,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:28,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:29,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:30,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:31,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:32,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:33,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:25:33,446 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:25:33,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:34,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:34,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:35,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:36,282 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:25:36,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:37,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:38,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:39,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:39,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:40,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:41,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:42,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:43,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:44,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:45,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:45,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:46,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:47,445 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:25:47,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:48,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:25:48,449 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:25:48,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:49,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:50,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:50,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:51,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:52,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:25:52,326 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:25:52,326 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:25:52,328 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:25:52,734 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:53,300 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:25:53,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:54,301 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:25:54,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:55,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:56,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:56,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:25:57,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:58,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:25:59,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:00,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:01,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:02,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:02,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:03,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:26:03,449 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:26:03,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:04,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:05,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:06,309 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:26:06,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:07,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:07,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:08,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:09,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:10,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:11,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:12,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:13,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:13,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:14,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:15,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:16,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:17,447 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:26:17,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:18,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:26:18,453 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:26:18,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:18,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:19,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:20,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:21,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:22,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:22,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:26:22,949 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:26:22,949 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:26:22,951 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:26:23,325 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:26:23,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:24,326 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:26:24,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:24,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:25,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:26,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:27,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:28,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:29,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:30,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:30,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:31,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:32,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:33,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:26:33,453 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:26:33,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:34,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:35,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:36,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:36,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:37,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:38,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:39,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:40,336 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:26:40,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:41,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:41,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:42,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:43,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:44,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:45,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:46,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:46,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:47,451 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:26:47,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:48,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:26:48,453 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:26:48,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:49,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:50,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:51,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:51,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:52,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:53,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:54,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:55,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:56,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:26:56,461 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:26:56,462 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:26:56,463 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:26:56,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:56,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:26:57,357 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:26:57,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:58,358 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:26:58,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:26:59,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:00,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:01,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:02,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:02,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:03,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:27:03,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:27:03,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:04,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:05,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:06,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:07,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:08,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:08,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:09,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:10,366 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:27:10,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:11,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:12,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:13,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:13,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:14,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:15,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:16,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:17,455 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:27:17,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:18,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:27:18,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:27:18,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:19,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:19,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:20,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:21,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:22,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:23,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:24,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:24,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:25,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:26,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:27,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:27:27,110 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:27:27,111 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:27:27,112 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:27:27,389 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:27:27,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:28,390 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:27:28,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:29,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:29,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:30,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:31,880 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:32,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:33,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:27:33,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:27:33,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:34,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:35,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:35,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:36,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:37,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:38,881 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:39,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:40,399 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:27:40,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:40,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:41,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:42,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:43,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:44,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:45,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:46,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:46,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:47,459 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:27:47,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:48,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:27:48,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:27:48,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:49,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:50,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:51,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:51,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:52,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:53,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:54,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:55,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:27:55,793 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:27:55,794 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:27:55,795 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:27:56,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:56,417 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:27:57,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:57,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:27:58,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:27:58,418 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:27:59,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:00,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:01,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:02,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:02,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:03,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:03,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:28:03,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:28:04,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:05,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:06,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:07,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:08,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:08,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:09,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:10,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:10,427 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:28:11,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:12,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:13,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:13,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:14,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:15,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:16,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:17,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:17,463 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:28:18,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:18,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:28:18,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:28:18,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:19,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:20,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:21,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:22,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:23,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:23,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:24,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:24,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:28:24,747 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:28:24,747 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:28:24,749 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:28:25,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:25,453 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:28:26,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:26,453 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:28:27,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:28,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:28,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:29,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:30,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:31,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:32,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:33,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:33,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:28:33,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:28:33,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:34,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:35,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:36,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:37,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:38,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:38,463 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:28:38,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:39,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:40,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:41,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:42,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:43,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:43,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:44,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:45,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:46,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:47,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:47,467 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:28:48,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:48,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:28:48,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:28:48,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:49,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:50,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:51,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:52,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:53,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:54,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:54,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:28:54,541 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:28:54,541 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:28:54,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:28:54,542 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:28:55,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:55,486 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:28:56,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:56,487 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:28:57,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:58,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:59,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:28:59,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:00,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:01,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:02,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:03,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:03,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:29:03,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:29:04,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:04,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:05,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:06,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:07,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:08,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:09,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:10,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:10,497 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:29:10,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:11,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:12,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:13,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:14,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:15,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:15,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:16,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:17,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:17,471 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:29:18,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:18,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:29:18,454 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:29:19,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:20,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:21,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:21,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:22,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:23,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:24,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:25,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:26,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:26,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:27,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:29:27,122 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:29:27,122 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:29:27,124 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:29:27,521 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:29:27,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:28,522 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:29:28,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:29,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:30,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:31,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:32,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:32,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:33,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:29:33,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:29:33,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:34,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:35,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:36,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:37,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:38,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:38,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:39,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:40,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:41,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:42,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:42,531 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:29:43,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:44,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:44,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:45,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:46,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:47,475 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:29:47,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:48,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:29:48,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:29:48,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:49,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:49,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:50,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:51,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:52,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:53,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:54,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:54,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:29:55,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:56,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:57,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:29:57,014 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:29:57,014 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:29:57,016 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:29:57,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:57,548 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:29:58,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:58,549 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:29:59,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:29:59,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:00,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:01,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:02,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:03,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:30:03,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:30:03,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:04,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:05,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:05,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:06,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:07,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:08,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:09,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:10,557 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:30:10,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:11,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:11,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:12,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:13,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:14,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:15,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:16,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:16,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:17,480 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:30:17,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:18,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:30:18,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:30:18,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:19,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:20,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:21,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:22,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:22,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:23,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:24,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:25,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:26,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:27,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:27,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:28,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:30:28,024 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:30:28,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:30:28,026 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:30:28,582 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:30:28,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:29,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:30,584 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:30:30,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:31,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:32,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:33,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:30:33,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:30:33,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:33,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:34,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:35,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:36,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:37,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:38,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:38,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:39,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:40,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:41,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:42,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:43,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:43,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:44,593 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:30:44,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:45,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:46,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:47,483 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:30:47,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:48,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:30:48,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:30:48,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:49,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:49,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:50,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:51,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:52,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:53,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:54,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:54,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:30:55,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:56,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:57,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:58,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:59,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:30:59,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:00,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:31:00,659 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:31:00,660 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:31:00,661 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:31:01,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:01,614 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:31:01,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:02,615 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:31:02,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:03,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:31:03,455 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:31:03,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:04,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:05,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:05,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:06,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:07,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:08,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:09,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:10,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:11,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:11,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:12,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:13,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:14,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:15,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:16,627 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:31:16,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:17,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:17,486 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:31:17,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:18,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:31:18,456 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:31:18,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:19,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:20,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:21,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:22,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:22,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:23,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:24,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:25,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:26,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:27,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:28,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:28,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:29,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:30,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:31,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:32,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:33,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:31:33,456 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:31:33,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:31:33,548 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:31:33,549 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:31:33,550 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:31:33,654 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:31:33,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:33,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:34,655 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:31:34,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:35,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:36,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:37,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:38,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:39,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:39,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:40,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:41,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:42,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:43,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:44,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:45,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:45,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:46,663 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:31:46,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:47,490 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:31:47,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:48,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:31:48,456 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:31:48,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:49,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:50,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:50,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:51,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:52,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:53,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:54,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:55,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:56,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:31:56,965 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:57,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:58,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:31:59,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:00,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:01,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:01,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:02,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:32:02,627 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:32:02,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:32:02,630 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:32:02,681 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:32:03,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:03,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:32:03,456 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:32:04,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:04,682 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:32:05,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:06,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:07,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:07,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:08,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:09,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:10,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:11,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:12,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:13,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:13,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:14,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:15,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:16,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:16,690 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:32:17,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:17,494 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:32:18,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:18,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:32:18,456 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:32:18,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:19,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:20,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:21,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:22,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:23,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:23,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:24,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:25,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:26,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:27,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:28,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:28,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:29,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:30,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:31,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:32,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:32,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:32:32,468 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:32:32,468 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:32:32,470 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:32:32,716 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:32:33,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:33,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:32:33,456 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:32:34,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:34,718 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:32:34,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:35,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:36,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:37,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:38,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:39,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:40,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:40,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:41,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:42,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:43,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:44,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:45,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:46,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:46,096 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:46,727 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:32:47,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:47,498 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:32:48,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:48,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:32:48,456 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:32:49,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:50,093 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:51,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:51,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:52,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:53,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:54,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:55,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:56,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:56,967 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:32:57,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:58,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:32:59,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:00,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:01,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:01,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:02,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:03,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:03,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:33:03,463 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:33:04,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:05,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:05,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:33:05,141 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:33:05,141 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:33:05,143 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:33:05,747 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:33:06,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:06,748 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:33:07,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:07,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:08,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:09,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:10,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:11,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:12,107 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:12,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:13,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:14,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:15,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:16,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:17,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:17,503 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:33:18,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:18,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:33:18,460 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:33:18,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:19,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:20,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:20,758 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:33:21,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:22,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:23,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:23,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:24,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:25,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:26,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:27,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:28,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:28,968 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:29,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:30,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:31,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:32,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:33,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:33,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:33:33,462 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:33:33,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:34,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:35,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:36,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:33:36,086 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:33:36,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:33:36,088 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:33:36,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:36,783 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:33:36,783 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:33:37,495 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:38,495 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:38,784 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:33:39,495 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:39,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:40,495 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:41,495 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:42,495 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:43,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:44,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:45,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:45,972 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:46,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:47,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:47,507 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:33:48,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:33:48,462 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:33:48,502 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:49,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:50,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:50,793 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:33:51,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:51,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:52,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:53,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:54,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:55,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:56,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:57,081 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:33:57,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:58,497 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:33:59,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:00,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:01,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:02,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:02,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:03,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:34:03,463 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:34:03,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:04,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:04,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:34:04,688 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:34:04,689 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:34:04,690 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:34:04,812 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:34:05,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:06,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:06,813 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:34:07,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:07,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:08,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:09,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:10,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:11,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:12,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:13,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:13,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:14,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:15,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:16,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:17,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:17,511 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:34:18,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:34:18,463 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:34:18,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:18,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:18,820 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:34:19,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:20,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:21,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:22,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:23,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:24,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:24,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:25,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:26,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:27,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:28,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:29,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:29,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:30,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:31,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:32,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:33,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:34:33,463 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:34:33,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:33,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:34:33,812 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:34:33,812 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:34:33,814 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:34:33,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:34:34,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:34,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:34,843 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:34:35,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:36,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:37,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:38,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:39,515 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:40,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:40,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:41,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:42,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:43,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:44,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:45,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:46,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:46,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:47,515 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:34:47,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:48,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:34:48,464 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:34:48,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:48,851 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:34:49,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:50,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:51,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:51,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:52,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:53,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:54,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:55,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:56,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:57,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:34:57,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:58,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:34:59,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:00,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:01,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:02,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:02,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:03,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:35:03,472 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:35:03,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:04,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:05,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:05,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:35:05,584 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:35:05,585 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:35:05,586 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:35:05,872 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:35:06,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:06,873 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:35:07,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:07,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:08,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:09,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:10,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:11,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:12,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:13,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:13,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:14,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:15,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:16,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:17,517 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:35:17,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:18,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:35:18,467 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:35:18,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:18,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:18,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:35:19,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:20,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:21,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:22,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:23,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:24,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:24,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:25,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:26,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:27,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:28,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:29,222 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:29,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:30,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:31,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:32,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:33,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:35:33,475 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:35:33,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:34,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:34,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:35,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:35:35,215 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:35:35,215 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:35:35,216 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:35:35,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:35,901 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:35:36,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:36,901 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:35:37,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:38,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:39,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:39,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:40,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:41,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:42,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:43,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:44,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:45,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:45,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:46,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:47,522 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:35:47,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:48,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:35:48,472 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:35:48,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:48,909 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:35:49,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:50,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:51,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:51,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:52,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:53,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:54,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:55,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:56,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:35:56,730 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:57,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:58,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:35:59,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:00,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:01,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:01,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:02,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:03,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:36:03,027 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:36:03,027 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:36:03,029 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:36:03,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:36:03,472 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:36:03,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:03,941 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:36:04,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:04,942 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:36:05,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:06,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:07,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:07,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:08,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:09,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:10,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:11,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:12,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:13,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:13,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:14,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:15,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:16,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:16,950 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:36:17,526 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:36:17,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:18,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:36:18,473 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:36:18,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:18,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:19,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:20,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:21,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:22,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:23,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:24,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:24,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:25,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:26,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:27,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:28,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:29,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:29,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:30,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:31,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:32,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:33,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:36:33,482 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:36:33,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:33,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:36:33,986 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:36:33,986 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:36:33,988 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:36:34,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:34,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:34,972 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:36:34,973 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:36:35,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:36,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:37,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:38,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:39,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:40,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:40,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:41,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:42,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:43,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:44,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:45,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:46,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:46,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:46,981 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:36:47,530 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:36:47,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:48,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:36:48,476 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:36:48,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:49,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:50,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:51,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:51,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:52,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:53,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:54,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:55,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:56,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:36:56,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:57,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:58,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:36:59,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:00,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:01,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:02,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:02,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:03,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:37:03,477 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:37:03,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:03,960 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:37:03,962 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:37:03,962 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:37:03,964 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:37:03,998 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:37:04,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:04,999 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:37:05,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:06,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:07,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:07,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:08,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:09,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:10,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:11,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:12,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:12,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:13,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:14,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:15,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:16,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:17,534 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:37:17,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:17,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:18,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:37:18,612 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:37:18,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:19,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:20,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:21,008 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:37:21,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:22,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:23,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:23,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:24,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:25,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:26,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:27,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:28,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:28,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:29,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:30,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:31,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:32,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:33,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:33,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:37:33,574 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:37:33,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:34,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:35,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:35,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:37:35,790 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:37:35,790 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:37:35,792 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:37:36,036 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:37:36,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:37,037 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:37:37,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:38,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:39,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:39,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:40,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:41,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:42,771 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:43,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:44,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:44,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:45,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:46,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:47,538 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:37:47,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:48,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:37:48,574 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:37:48,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:49,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:49,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:50,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:51,045 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:37:51,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:52,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:53,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:54,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:37:54,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:55,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:56,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:57,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:58,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:37:59,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:00,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:00,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:01,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:02,778 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:03,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:38:03,581 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:38:03,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:04,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:05,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:05,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:06,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:07,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:38:07,618 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:38:07,619 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:38:07,620 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:38:08,061 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:38:08,069 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:09,061 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:38:09,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:10,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:11,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:11,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:12,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:13,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:14,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:15,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:16,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:17,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:17,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:17,542 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:38:18,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:18,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:38:18,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:38:19,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:20,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:21,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:22,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:22,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:23,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:23,072 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:38:24,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:25,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:26,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:27,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:28,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:28,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:29,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:30,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:31,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:32,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:33,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:33,443 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:33,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:38:33,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:38:34,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:35,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:36,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:37,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:38,072 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:38,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:38,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:38:38,869 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:38:38,869 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:38:38,870 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:38:39,090 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:38:39,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:40,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:41,091 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:38:41,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:42,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:43,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:44,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:44,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:45,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:46,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:47,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:47,546 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:38:48,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:48,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:38:48,578 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:38:49,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:50,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:50,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:51,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:52,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:53,098 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:38:53,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:54,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:55,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:55,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:38:56,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:57,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:58,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:38:59,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:00,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:01,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:01,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:02,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:03,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:03,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:39:03,584 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:39:04,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:05,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:06,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:06,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:07,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:07,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:39:07,942 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:39:07,943 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:39:07,944 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:39:08,117 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:39:08,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:09,118 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:39:09,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:10,359 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:11,120 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:39:11,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:12,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:12,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:13,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:14,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:15,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:16,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:17,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:17,550 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:39:17,556 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:18,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:18,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:39:18,583 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:39:19,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:20,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:21,126 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:39:21,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:22,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:23,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:23,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:24,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:25,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:26,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:27,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:28,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:28,446 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:29,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:30,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:31,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:32,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:33,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:33,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:33,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:39:33,589 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:39:34,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:35,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:36,423 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:37,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:38,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:39:38,159 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:39:38,160 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:39:38,161 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:39:38,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:38,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:39,143 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:39:39,144 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:39:39,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:40,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:41,144 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:39:41,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:42,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:43,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:44,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:44,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:45,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:46,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:47,554 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:39:47,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:48,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:48,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:39:48,589 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:39:49,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:50,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:50,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:51,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:52,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:53,152 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:39:53,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:54,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:55,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:39:55,597 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:56,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:57,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:58,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:39:59,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:00,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:01,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:01,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:02,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:03,594 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:40:03,595 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:40:03,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:04,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:05,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:06,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:06,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:07,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:08,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:08,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:40:08,725 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:40:08,726 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:40:08,728 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:40:09,168 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:40:09,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:10,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:11,169 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:40:11,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:12,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:12,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:13,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:14,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:15,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:16,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:17,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:17,557 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:40:17,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:18,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:40:18,589 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:40:18,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:19,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:20,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:21,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:22,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:23,177 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:40:23,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:23,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:24,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:25,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:26,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:27,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:28,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:28,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:29,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:30,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:31,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:32,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:33,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:33,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:40:33,599 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:40:33,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:34,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:35,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:36,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:37,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:38,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:38,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:38,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:40:38,832 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:40:38,832 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:40:38,833 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:40:39,198 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:40:39,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:40,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:41,199 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:40:41,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:42,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:43,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:44,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:44,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:45,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:46,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:47,561 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:40:47,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:48,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:40:48,593 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:40:48,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:49,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:50,452 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:50,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:51,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:52,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:53,206 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:40:53,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:54,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:55,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:40:55,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:56,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:57,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:58,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:40:59,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:00,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:00,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:01,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:02,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:03,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:41:03,602 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:41:03,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:04,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:05,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:05,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:06,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:07,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:07,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:41:07,639 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:41:07,639 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:41:07,641 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:41:08,225 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:41:08,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:09,226 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:41:09,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:10,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:10,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:11,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:12,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:13,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:14,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:15,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:16,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:16,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:17,564 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:41:17,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:18,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:41:18,595 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:41:18,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:19,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:20,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:21,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:22,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:22,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:23,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:24,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:25,237 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:41:25,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:26,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:27,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:27,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:28,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:29,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:30,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:31,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:32,456 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:32,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:33,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:41:33,595 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:41:33,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:34,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:35,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:36,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:37,459 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:37,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:38,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:39,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:40,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:40,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:41:40,723 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:41:40,723 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:41:40,725 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:41:41,254 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:41:41,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:42,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:43,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:43,255 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:41:43,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:44,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:45,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:46,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:47,567 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:41:47,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:48,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:48,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:41:48,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:41:48,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:49,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:50,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:51,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:52,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:53,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:54,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:54,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:55,263 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:41:55,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:56,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:57,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:58,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:41:59,458 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:41:59,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:00,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:01,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:02,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:03,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:42:03,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:42:03,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:04,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:05,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:05,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:06,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:07,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:08,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:09,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:10,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:10,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:11,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:42:11,410 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:42:11,411 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:42:11,412 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:42:11,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:12,283 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:42:12,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:13,284 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:42:13,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:14,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:15,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:16,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:16,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:17,571 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:42:17,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:18,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:42:18,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:42:18,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:19,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:20,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:21,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:22,463 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:22,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:23,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:24,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:25,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:26,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:27,292 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:42:27,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:28,324 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:28,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:29,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:30,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:31,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:32,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:33,464 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:33,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:42:33,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:42:33,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:34,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:35,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:36,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:37,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:38,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:38,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:39,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:40,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:41,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:42,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:43,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:43,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:42:43,953 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:42:43,954 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:42:43,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:43,956 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:42:44,311 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:42:44,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:45,312 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:42:45,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:46,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:47,574 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:42:47,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:48,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:42:48,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:42:48,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:49,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:49,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:50,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:51,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:52,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:53,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:54,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:55,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:42:55,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:56,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:57,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:58,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:42:59,321 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:42:59,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:00,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:00,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:01,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:02,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:03,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:43:03,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:43:03,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:04,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:05,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:06,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:06,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:07,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:08,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:09,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:10,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:11,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:11,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:12,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:13,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:14,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:15,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:43:15,388 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:43:15,389 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:43:15,390 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:43:15,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:16,341 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:43:16,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:16,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:17,341 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:43:17,578 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:43:17,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:18,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:43:18,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:43:18,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:19,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:20,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:21,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:22,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:22,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:23,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:24,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:25,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:26,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:27,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:28,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:28,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:30,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:31,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:31,350 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:43:32,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:33,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:33,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:43:33,596 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:43:33,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:34,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:35,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:36,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:37,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:38,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:39,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:39,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:40,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:41,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:42,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:43,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:44,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:44,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:45,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:46,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:47,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:47,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:43:47,501 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:43:47,502 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:43:47,504 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:43:47,582 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:43:48,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:48,376 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:43:48,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:43:48,597 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:43:49,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:49,377 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:43:49,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:50,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:51,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:52,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:53,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:54,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:54,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:43:55,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:56,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:57,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:58,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:43:59,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:00,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:00,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:01,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:01,386 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:44:02,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:03,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:03,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:44:03,597 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:44:04,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:05,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:05,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:06,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:07,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:08,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:09,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:10,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:11,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:11,474 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:12,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:13,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:14,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:15,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:16,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:16,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:17,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:17,584 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:44:17,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:44:17,894 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:44:17,894 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:44:17,896 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:44:18,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:18,407 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:44:18,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:44:18,597 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:44:19,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:19,408 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:44:20,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:21,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:21,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:22,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:23,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:24,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:25,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:26,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:26,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:27,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:28,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:29,307 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:30,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:31,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:31,416 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:44:32,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:32,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:33,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:33,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:44:33,597 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:44:34,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:35,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:36,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:37,363 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:37,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:38,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:39,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:40,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:41,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:42,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:43,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:43,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:44,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:45,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:46,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:47,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:47,588 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:44:48,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:48,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:44:48,431 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:44:48,431 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:44:48,433 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:44:48,439 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:44:48,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:48,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:44:48,814 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:44:49,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:49,440 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:44:50,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:51,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:51,441 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:44:52,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:53,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:54,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:54,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:44:55,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:56,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:57,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:58,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:59,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:44:59,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:00,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:01,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:01,448 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:45:02,371 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:03,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:03,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:45:03,812 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:45:04,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:05,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:05,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:06,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:07,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:08,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:09,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:10,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:10,488 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:11,372 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:12,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:13,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:14,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:15,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:15,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:16,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:17,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:17,445 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:45:17,446 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:45:17,446 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:45:17,447 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:45:17,467 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:45:17,592 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:45:18,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:18,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:45:18,812 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:45:19,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:19,469 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:45:20,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:20,493 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:21,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:22,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:23,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:24,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:25,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:25,493 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:26,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:27,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:28,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:29,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:30,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:31,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:31,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:31,476 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:45:32,465 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:33,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:33,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:45:33,812 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:45:34,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:35,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:36,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:36,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:37,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:38,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:39,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:40,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:41,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:42,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:42,494 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:43,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:44,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:45,480 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:46,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:47,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:47,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:47,597 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:45:48,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:45:48,411 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:45:48,412 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:45:48,413 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:45:48,495 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:45:48,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:48,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:45:48,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:45:49,497 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:45:49,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:50,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:51,498 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:45:51,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:52,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:52,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:53,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:54,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:55,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:56,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:57,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:45:57,557 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:58,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:45:59,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:00,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:01,504 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:46:01,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:02,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:02,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:03,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:03,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:46:03,819 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:46:04,558 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:05,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:06,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:07,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:08,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:08,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:09,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:10,559 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:11,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:12,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:13,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:13,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:14,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:15,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:16,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:17,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:46:17,041 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:46:17,042 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:46:17,043 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:46:17,525 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:46:17,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:17,601 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:46:18,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:18,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:18,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:46:18,819 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:46:19,526 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:46:19,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:20,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:21,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:22,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:23,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:24,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:24,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:25,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:26,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:27,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:28,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:29,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:29,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:30,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:31,535 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:46:31,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:32,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:33,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:33,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:46:33,819 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:46:34,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:35,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:35,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:36,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:37,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:38,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:39,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:40,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:40,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:41,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:42,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:43,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:44,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:45,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:45,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:46,023 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:46:46,024 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:46:46,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:46:46,026 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:46:46,557 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:46:46,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:47,558 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:46:47,605 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:46:47,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:48,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:48,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:46:48,819 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:46:49,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:50,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:51,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:51,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:52,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:53,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:54,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:55,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:56,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:57,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:46:57,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:58,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:46:59,567 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:46:59,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:00,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:01,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:02,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:03,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:03,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:03,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:47:03,819 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:47:04,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:05,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:06,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:07,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:08,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:09,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:09,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:10,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:11,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:12,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:13,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:14,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:14,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:47:14,523 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:47:14,523 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:47:14,525 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:47:14,578 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:47:14,934 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:15,579 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:47:15,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:16,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:17,580 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:47:17,609 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:47:17,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:18,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:18,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:47:18,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:47:19,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:20,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:20,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:21,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:22,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:23,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:24,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:25,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:26,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:26,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:27,586 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:47:27,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:28,666 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:29,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:30,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:31,504 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:31,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:32,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:33,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:33,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:47:33,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:47:34,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:35,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:36,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:36,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:37,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:38,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:39,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:40,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:41,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:41,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:47:41,660 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:47:41,660 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:47:41,662 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:47:42,070 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:42,605 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:47:42,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:43,606 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:47:43,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:44,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:45,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:46,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:47,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:47,613 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:47:47,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:48,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:48,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:47:48,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:47:49,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:50,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:51,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:52,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:53,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:53,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:54,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:55,613 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:47:55,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:56,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:57,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:58,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:47:58,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:47:59,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:00,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:01,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:02,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:03,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:03,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:48:03,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:48:03,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:04,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:05,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:06,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:07,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:08,510 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:08,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:09,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:10,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:11,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:48:11,255 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:48:11,256 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:48:11,257 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:48:11,635 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:48:11,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:12,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:13,636 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:48:13,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:13,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:14,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:15,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:16,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:17,618 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:48:17,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:18,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:48:18,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:48:18,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:18,988 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:19,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:20,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:21,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:22,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:23,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:24,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:24,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:25,643 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:48:25,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:26,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:27,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:28,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:29,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:29,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:30,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:31,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:32,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:33,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:48:33,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:48:33,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:34,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:35,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:35,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:36,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:37,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:38,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:39,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:40,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:41,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:48:41,277 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:48:41,278 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:48:41,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:41,280 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:48:41,669 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:48:41,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:42,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:43,670 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:48:43,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:44,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:45,860 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:46,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:46,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:47,621 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:48:47,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:48,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:48:48,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:48:48,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:49,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:50,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:51,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:52,516 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:52,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:53,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:54,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:55,678 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:48:55,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:56,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:57,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:48:57,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:58,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:48:59,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:00,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:01,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:02,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:03,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:03,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:49:03,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:49:03,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:04,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:05,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:06,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:07,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:08,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:08,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:09,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:10,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:11,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:12,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:49:12,742 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:49:12,742 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:49:12,744 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:49:13,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:13,714 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:49:13,715 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:49:14,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:14,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:15,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:15,715 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:49:16,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:17,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:17,625 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:49:18,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:18,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:49:18,820 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:49:19,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:19,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:20,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:21,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:22,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:23,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:24,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:25,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:25,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:26,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:27,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:27,724 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:49:28,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:29,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:30,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:30,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:31,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:32,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:33,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:33,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:49:33,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:49:34,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:35,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:36,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:36,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:37,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:38,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:39,210 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:40,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:41,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:41,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:42,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:43,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:43,301 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:49:43,303 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:49:43,303 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:49:43,304 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:49:43,748 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:49:44,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:45,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:45,749 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:49:46,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:47,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:47,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:47,629 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:49:48,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:48,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:49:48,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:49:49,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:50,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:51,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:52,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:53,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:53,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:54,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:55,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:56,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:57,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:57,757 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:49:58,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:49:58,716 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:49:59,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:00,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:01,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:02,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:03,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:03,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:50:03,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:50:03,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:04,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:05,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:06,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:07,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:08,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:09,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:09,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:10,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:11,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:12,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:12,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:50:12,529 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:50:12,529 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:50:12,530 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:50:12,783 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:50:13,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:13,784 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:50:14,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:14,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:15,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:16,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:17,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:17,633 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:50:18,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:18,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:50:18,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:50:19,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:20,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:20,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:21,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:22,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:23,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:24,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:25,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:26,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:26,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:27,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:27,792 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:50:28,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:29,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:30,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:31,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:31,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:32,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:33,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:33,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:50:33,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:50:34,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:35,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:36,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:37,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:37,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:38,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:39,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:40,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:41,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:42,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:42,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:42,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:50:42,952 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:50:42,953 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:50:42,954 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:50:43,368 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:43,814 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:50:43,815 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:50:44,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:45,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:45,817 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:50:46,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:47,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:47,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:47,637 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:50:48,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:48,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:50:48,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:50:49,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:50,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:51,366 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:52,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:53,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:53,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:50:54,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:55,367 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:56,495 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:57,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:57,824 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:50:58,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:59,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:50:59,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:00,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:01,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:02,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:03,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:03,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:51:03,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:51:04,453 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:04,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:05,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:06,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:07,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:08,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:09,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:09,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:10,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:11,457 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:12,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:51:12,391 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:51:12,391 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:51:12,393 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:51:12,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:12,850 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:51:13,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:13,851 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:51:14,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:14,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:15,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:16,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:17,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:17,641 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:51:18,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:18,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:51:18,821 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:51:19,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:20,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:20,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:21,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:22,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:23,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:24,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:25,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:26,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:26,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:27,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:27,860 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:51:28,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:29,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:30,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:31,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:31,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:32,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:33,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:33,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:51:33,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:51:34,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:35,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:36,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:37,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:37,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:38,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:39,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:40,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:41,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:42,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:43,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:51:43,079 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:51:43,079 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:51:43,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:43,080 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:51:43,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:43,874 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:51:43,875 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:51:44,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:45,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:45,876 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:51:46,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:47,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:47,644 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:51:48,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:48,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:48,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:51:48,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:51:49,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:50,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:51,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:52,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:53,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:54,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:54,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:51:55,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:56,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:57,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:57,885 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:51:58,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:59,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:51:59,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:00,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:01,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:02,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:03,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:03,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:52:03,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:52:04,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:05,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:05,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:06,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:07,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:08,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:09,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:10,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:10,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:11,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:12,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:13,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:14,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:52:14,444 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:52:14,444 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:52:14,446 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:52:14,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:14,901 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:52:15,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:15,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:15,903 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:52:16,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:17,647 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:52:17,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:18,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:52:18,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:52:18,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:19,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:20,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:21,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:21,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:22,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:23,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:24,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:25,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:26,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:27,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:27,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:27,914 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:52:28,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:29,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:30,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:31,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:32,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:32,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:33,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:52:33,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:52:33,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:34,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:35,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:36,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:37,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:37,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:38,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:39,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:40,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:41,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:42,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:52:42,172 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:52:42,173 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:52:42,174 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:52:42,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:42,934 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:52:43,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:43,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:43,935 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:52:44,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:45,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:46,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:47,651 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:52:47,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:48,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:48,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:52:48,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:52:48,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:49,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:50,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:51,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:52,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:53,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:53,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:54,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:55,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:56,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:57,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:57,945 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:52:58,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:52:59,707 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:52:59,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:00,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:01,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:02,855 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:03,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:53:03,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:53:03,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:04,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:04,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:05,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:06,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:07,856 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:08,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:09,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:09,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:10,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:11,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:12,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:13,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:14,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:53:14,449 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:53:14,450 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:53:14,451 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:53:14,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:14,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:14,961 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:53:15,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:15,962 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:53:16,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:17,655 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:53:17,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:18,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:53:18,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:53:18,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:19,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:20,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:20,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:21,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:22,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:23,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:24,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:25,864 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:26,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:26,924 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:27,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:27,970 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:53:28,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:29,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:30,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:31,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:31,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:32,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:33,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:53:33,822 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:53:33,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:34,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:35,970 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:36,970 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:37,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:37,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:38,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:39,978 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:40,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:41,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:42,979 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:43,858 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:43,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:44,265 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:53:44,267 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:53:44,267 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:53:44,268 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:53:44,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:44,994 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:53:45,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:45,995 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:53:46,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:47,660 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:53:47,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:48,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:53:48,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:53:48,902 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:48,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:49,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:50,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:51,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:52,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:53,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:54,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:54,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:55,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:56,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:57,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:58,004 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:53:58,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:53:59,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:53:59,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:00,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:01,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:02,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:03,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:54:03,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:54:03,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:04,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:05,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:05,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:06,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:07,982 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:08,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:09,984 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:10,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:10,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:11,987 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:12,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:54:12,848 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:54:12,849 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:54:12,850 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:54:13,024 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:54:13,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:13,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:14,025 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:54:14,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:15,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:16,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:16,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:17,663 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:54:17,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:18,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:54:18,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:54:18,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:19,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:20,990 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:21,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:22,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:22,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:23,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:24,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:25,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:26,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:27,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:28,033 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:54:28,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:28,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:29,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:30,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:31,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:32,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:33,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:54:33,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:54:33,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:33,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:34,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:35,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:36,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:38,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:38,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:39,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:39,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:40,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:41,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:42,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:43,347 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:54:43,348 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:54:43,349 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:54:43,350 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:54:43,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:44,056 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:54:44,057 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:54:44,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:45,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:45,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:46,058 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:54:46,996 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:47,667 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:54:47,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:48,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:54:48,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:54:48,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:49,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:50,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:51,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:51,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:52,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:53,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:54,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:55,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:56,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:54:57,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:58,067 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:54:58,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:54:59,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:00,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:01,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:01,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:02,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:03,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:03,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:55:03,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:55:04,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:05,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:06,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:07,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:07,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:08,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:09,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:10,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:11,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:12,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:12,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:13,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:14,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:14,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:55:14,766 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:55:14,766 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:55:14,768 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:55:15,095 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:55:15,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:16,096 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:55:16,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:17,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:17,671 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:55:18,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:18,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:18,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:55:18,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:55:19,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:20,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:21,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:22,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:23,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:23,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:24,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:25,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:26,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:27,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:28,667 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:29,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:29,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:30,105 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:55:30,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:31,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:32,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:33,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:33,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:55:33,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:55:34,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:34,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:35,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:36,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:37,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:38,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:39,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:39,908 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:40,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:41,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:42,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:43,314 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:44,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:44,917 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:45,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:46,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:47,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:47,676 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:55:47,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:55:47,724 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:55:47,725 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:55:47,726 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:55:48,125 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:55:48,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:48,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:55:48,823 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:55:49,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:50,126 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:55:50,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:50,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:51,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:52,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:53,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:54,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:55,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:56,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:56,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:55:57,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:58,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:55:59,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:00,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:01,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:02,134 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:56:02,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:02,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:03,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:03,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:56:03,824 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:56:04,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:05,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:06,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:07,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:07,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:08,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:09,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:10,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:11,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:12,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:12,921 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:13,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:14,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:15,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:16,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:17,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:17,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:56:17,639 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:56:17,639 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:56:17,640 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:56:17,678 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:56:18,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:18,161 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:56:18,161 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:56:18,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:18,826 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:56:18,826 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:56:19,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:20,162 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:56:20,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:21,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:22,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:23,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:23,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:24,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:25,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:26,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:27,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:28,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:28,916 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:29,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:30,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:31,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:32,170 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:56:32,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:33,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:33,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:56:33,824 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:56:33,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:34,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:35,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:36,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:37,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:38,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:39,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:39,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:40,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:41,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:42,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:43,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:44,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:44,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:45,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:46,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:47,331 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:47,679 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:56:48,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:56:48,078 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:56:48,079 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:56:48,080 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:56:48,189 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:56:48,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:48,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:56:48,824 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:56:49,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:49,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:50,191 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:56:50,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:51,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:52,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:53,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:54,461 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:55,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:55,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:56:56,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:57,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:58,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:56:59,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:00,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:01,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:01,622 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:02,199 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:57:02,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:03,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:03,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:57:03,824 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:57:04,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:05,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:06,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:06,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:07,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:08,522 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:09,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:10,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:11,523 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:11,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:12,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:13,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:14,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:15,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:16,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:16,936 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:17,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:17,687 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:57:17,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:57:17,872 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:57:17,872 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:57:17,874 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:57:18,227 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:57:18,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:18,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:57:18,824 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:57:19,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:20,229 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:57:20,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:21,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:22,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:22,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:23,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:24,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:25,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:26,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:27,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:28,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:28,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:29,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:30,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:31,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:32,236 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:57:32,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:33,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:33,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:57:33,825 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:57:33,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:34,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:35,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:36,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:37,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:38,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:38,992 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:39,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:40,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:41,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:42,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:43,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:43,997 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:44,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:45,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:46,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:47,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:47,691 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:57:47,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:57:47,794 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:57:47,794 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:57:47,797 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:57:48,253 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:57:48,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:48,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:57:48,825 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:57:49,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:49,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:50,254 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:57:50,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:51,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:52,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:53,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:54,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:54,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:57:55,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:56,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:57,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:58,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:57:59,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:00,260 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:58:00,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:00,993 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:01,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:02,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:03,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:03,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:58:03,825 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:58:04,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:05,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:05,994 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:06,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:07,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:08,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:09,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:10,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:11,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:11,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:12,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:13,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:14,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:15,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:16,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:16,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:58:16,973 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:58:16,973 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:58:16,974 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:58:17,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:17,278 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:58:17,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:17,695 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:58:18,279 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:58:18,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:18,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:58:18,825 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:58:19,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:20,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:21,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:22,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:22,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:23,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:24,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:25,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:26,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:27,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:28,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:28,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:29,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:30,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:31,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:32,288 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:58:32,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:33,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:33,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:58:33,825 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:58:34,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:34,991 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:35,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:36,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:37,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:38,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:39,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:39,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:40,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:41,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:42,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:43,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:44,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:45,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:45,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:46,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:47,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:58:47,273 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:58:47,273 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:58:47,275 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:58:47,309 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:58:47,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:47,699 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:58:48,310 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:58:48,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:48,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:58:48,825 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:58:49,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:50,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:51,674 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:51,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:52,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:53,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:54,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:55,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:56,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:56,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:58:57,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:58,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:58:59,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:00,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:01,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:02,319 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:59:02,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:02,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:03,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:03,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:59:03,825 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:59:04,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:05,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:06,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:07,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:07,999 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:08,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:09,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:10,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:11,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:12,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:13,000 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:13,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:14,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:15,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:16,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:17,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:17,704 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:59:18,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:18,710 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:18,830 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:59:18,831 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:59:19,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:19,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:59:19,765 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:59:19,765 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:59:19,767 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:59:20,341 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:59:20,341 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:59:20,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:21,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:22,343 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:59:22,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:23,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:24,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:24,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:25,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:26,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:27,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:28,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:29,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:30,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:30,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:31,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:32,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:33,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:33,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:59:33,829 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:59:34,350 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:59:34,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:35,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:35,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:36,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:37,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:38,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:39,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:40,171 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:40,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:41,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:42,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:43,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:44,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:45,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:45,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:46,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:47,692 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:47,711 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 13:59:48,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:48,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 13:59:48,827 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 13:59:49,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 13:59:49,645 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 13:59:49,646 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 13:59:49,648 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 13:59:50,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:50,368 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 13:59:50,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:51,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:51,369 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:59:51,785 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:52,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:53,371 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 13:59:53,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:54,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:55,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:56,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:57,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 13:59:57,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:58,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 13:59:59,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:00,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:01,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:02,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:02,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:03,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:03,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:00:03,827 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:00:04,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:05,380 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:00:05,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:06,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:07,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:08,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:08,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:09,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:10,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:11,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:12,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:13,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:13,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:14,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:15,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:16,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:17,712 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:00:17,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:18,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:18,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:18,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:00:18,827 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:00:19,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:19,977 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:00:19,979 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:00:19,979 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:00:19,984 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:00:20,399 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:00:20,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:21,400 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:00:21,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:22,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:23,401 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:00:23,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:24,385 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:24,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:25,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:26,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:27,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:28,795 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:29,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:30,382 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:30,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:31,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:32,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:33,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:33,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:00:33,828 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:00:34,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:35,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:35,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:36,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:37,410 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:00:37,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:38,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:39,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:40,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:40,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:41,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:42,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:43,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:44,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:45,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:46,386 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:46,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:47,717 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:00:47,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:48,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:48,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:00:48,831 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:00:49,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:50,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:51,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:51,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:51,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:00:51,860 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:00:51,861 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:00:51,865 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:00:52,428 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:00:52,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:53,429 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:00:53,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:54,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:55,431 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:00:55,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:56,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:00:56,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:57,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:58,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:00:59,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:00,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:01,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:01,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:02,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:03,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:03,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:01:03,831 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:01:04,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:05,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:06,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:06,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:07,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:08,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:09,440 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:01:09,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:10,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:11,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:12,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:12,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:13,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:14,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:15,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:16,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:17,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:17,718 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:01:17,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:18,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:18,840 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:01:18,844 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:01:19,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:20,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:21,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:22,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:22,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:23,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:24,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:01:24,544 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:01:24,544 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:01:24,547 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:01:24,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:25,463 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:01:25,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:26,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:27,464 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:01:27,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:28,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:28,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:29,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:30,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:31,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:32,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:33,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:33,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:01:33,837 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:01:33,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:34,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:35,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:36,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:37,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:38,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:38,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:39,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:40,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:41,473 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:01:41,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:42,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:43,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:44,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:44,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:45,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:46,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:47,722 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:01:47,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:48,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:01:48,837 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:01:48,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:49,954 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:49,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:50,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:51,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:52,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:53,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:54,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:55,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:01:55,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:56,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:01:56,323 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:01:56,323 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:01:56,328 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:01:56,489 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:01:56,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:57,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:58,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:01:59,491 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:01:59,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:00,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:00,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:01,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:02,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:03,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:02:03,838 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:02:03,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:04,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:05,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:06,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:06,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:08,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:09,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:09,498 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:02:10,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:11,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:11,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:12,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:13,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:14,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:15,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:16,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:17,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:17,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:17,726 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:02:18,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:18,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:02:18,838 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:02:19,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:20,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:21,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:22,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:22,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:23,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:24,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:25,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:25,692 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:02:25,694 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:02:25,694 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:02:25,696 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:02:26,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:26,520 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:02:27,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:27,521 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:02:28,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:28,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:29,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:30,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:31,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:32,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:33,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:33,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:33,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:02:33,838 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:02:34,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:35,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:36,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:37,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:38,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:38,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:39,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:40,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:41,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:41,531 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:02:42,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:43,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:44,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:44,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:45,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:46,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:47,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:47,730 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:02:48,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:48,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:02:48,838 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:02:49,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:49,925 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:50,174 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:51,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:52,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:53,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:54,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:55,179 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:55,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:02:56,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:57,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:02:57,130 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:02:57,130 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:02:57,135 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:02:57,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:57,554 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:02:58,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:59,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:02:59,555 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:03:00,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:00,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:01,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:02,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:03,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:03,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:03:03,838 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:03:04,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:05,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:05,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:06,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:07,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:08,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:09,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:10,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:11,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:11,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:11,563 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:03:12,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:13,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:14,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:15,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:16,272 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:16,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:17,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:17,733 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:03:18,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:18,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:03:18,854 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:03:19,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:20,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:21,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:21,402 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:22,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:23,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:24,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:24,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:03:24,421 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:03:24,422 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:03:24,423 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:03:24,583 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:03:25,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:26,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:26,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:27,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:27,586 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:03:28,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:29,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:30,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:31,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:32,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:32,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:33,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:33,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:03:33,844 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:03:34,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:35,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:36,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:37,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:38,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:38,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:39,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:40,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:41,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:41,594 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:03:42,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:43,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:43,411 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:44,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:45,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:46,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:47,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:47,737 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:03:48,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:48,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:48,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:03:48,844 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:03:49,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:50,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:51,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:52,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:53,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:54,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:54,431 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:03:55,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:56,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:03:56,016 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:03:56,016 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:03:56,018 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:03:56,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:56,613 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:03:57,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:57,614 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:03:58,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:59,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:03:59,615 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:04:00,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:00,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:01,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:02,427 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:03,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:03,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:04:03,844 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:04:04,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:05,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:06,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:06,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:07,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:08,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:09,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:10,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:11,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:11,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:12,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:13,428 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:13,624 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:04:14,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:15,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:16,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:17,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:17,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:17,741 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:04:18,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:18,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:04:18,844 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:04:19,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:20,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:21,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:22,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:23,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:23,429 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:24,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:25,433 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:26,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:27,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:28,420 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:28,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:29,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:30,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:04:30,093 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:04:30,094 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:04:30,095 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:04:30,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:30,646 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:04:31,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:31,647 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:04:32,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:33,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:33,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:33,648 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:04:33,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:04:33,845 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:04:34,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:35,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:36,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:37,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:38,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:39,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:39,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:40,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:41,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:42,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:43,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:44,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:45,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:45,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:45,656 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:04:46,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:47,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:47,745 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:04:48,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:48,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:04:48,845 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:04:49,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:50,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:50,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:51,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:52,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:53,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:54,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:55,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:56,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:04:56,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:57,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:58,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:04:59,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:00,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:01,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:01,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:02,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:03,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:05:03,077 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:05:03,078 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:05:03,079 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:05:03,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:03,675 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:05:03,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:05:03,852 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:05:04,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:05,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:05,677 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:05:06,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:06,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:07,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:08,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:09,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:10,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:11,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:12,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:12,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:13,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:14,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:15,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:16,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:17,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:17,693 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:05:17,750 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:05:17,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:18,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:18,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:05:18,851 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:05:19,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:20,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:21,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:22,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:23,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:23,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:24,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:25,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:26,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:27,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:28,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:28,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:29,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:30,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:31,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:32,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:33,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:33,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:33,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:05:33,789 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:05:33,789 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:05:33,790 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:05:34,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:05:34,204 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:05:34,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:34,716 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:05:35,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:35,716 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:05:36,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:37,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:38,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:39,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:39,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:40,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:41,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:42,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:43,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:44,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:45,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:45,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:46,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:47,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:47,754 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:05:48,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:48,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:05:48,932 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:05:49,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:49,725 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:05:50,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:50,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:51,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:52,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:53,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:54,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:55,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:56,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:05:56,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:57,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:58,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:05:59,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:00,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:01,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:01,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:02,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:03,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:03,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:06:03,932 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:06:04,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:06:04,261 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:06:04,262 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:06:04,263 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:06:04,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:04,751 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:06:05,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:05,752 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:06:06,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:07,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:07,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:07,753 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:06:08,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:09,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:10,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:11,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:12,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:13,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:13,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:14,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:15,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:16,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:17,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:17,758 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:06:17,765 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:06:18,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:18,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:18,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:06:18,932 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:06:19,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:20,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:21,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:22,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:23,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:24,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:24,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:25,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:26,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:27,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:28,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:29,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:29,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:30,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:31,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:32,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:33,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:33,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:06:33,933 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:06:34,094 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:06:34,096 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:06:34,096 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:06:34,098 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:06:34,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:34,783 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:06:35,498 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:35,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:35,783 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:06:36,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:37,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:38,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:39,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:40,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:40,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:41,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:42,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:43,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:44,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:45,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:46,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:46,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:47,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:47,762 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:06:47,791 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:06:48,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:48,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:06:48,932 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:06:49,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:50,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:51,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:51,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:52,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:53,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:54,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:55,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:56,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:57,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:06:57,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:58,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:06:59,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:00,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:01,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:02,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:02,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:03,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:03,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:07:03,880 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:07:03,881 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:07:03,882 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:07:04,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:07:04,292 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:07:04,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:04,808 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:07:05,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:05,809 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:07:06,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:07,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:07,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:08,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:09,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:10,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:11,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:12,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:12,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:13,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:14,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:15,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:16,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:17,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:17,766 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:07:17,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:18,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:19,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:07:19,021 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:07:19,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:20,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:21,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:21,818 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:07:22,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:23,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:23,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:24,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:25,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:26,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:27,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:28,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:28,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:29,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:30,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:31,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:32,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:33,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:33,736 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:34,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:07:34,032 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:07:34,728 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:35,732 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:36,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:37,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:38,633 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:38,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:39,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:07:39,540 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:07:39,540 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:07:39,542 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:07:39,841 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:07:39,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:40,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:41,842 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:07:41,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:42,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:43,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:44,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:44,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:45,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:46,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:47,770 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:07:47,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:48,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:49,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:07:49,024 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:07:49,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:50,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:50,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:51,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:52,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:53,850 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:07:53,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:54,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:55,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:07:55,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:56,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:57,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:58,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:07:59,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:00,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:00,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:01,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:02,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:03,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:04,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:08:04,024 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:08:04,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:05,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:05,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:06,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:07,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:08,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:09,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:10,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:11,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:08:11,045 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:08:11,046 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:08:11,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:11,048 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:08:11,872 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:08:11,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:12,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:13,874 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:08:13,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:14,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:15,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:16,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:16,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:17,774 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:08:17,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:18,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:19,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:08:19,024 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:08:19,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:20,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:21,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:22,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:22,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:23,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:24,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:25,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:08:25,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:26,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:27,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:27,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:28,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:29,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:30,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:31,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:32,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:32,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:33,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:34,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:08:34,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:08:34,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:35,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:36,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:37,646 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:37,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:38,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:39,970 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:40,971 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:41,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:08:41,763 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:08:41,763 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:08:41,765 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:08:41,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:08:42,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:43,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:43,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:43,907 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:08:44,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:45,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:46,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:47,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:47,778 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:08:48,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:48,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:49,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:08:49,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:08:49,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:50,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:51,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:52,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:53,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:54,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:54,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:08:55,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:55,914 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:08:56,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:57,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:58,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:59,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:08:59,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:00,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:01,376 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:02,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:03,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:04,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:09:04,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:09:04,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:05,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:05,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:06,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:07,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:08,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:09,384 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:10,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:10,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:11,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:11,721 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:09:11,722 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:09:11,722 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:09:11,723 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:09:11,931 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:09:12,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:13,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:13,932 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:09:14,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:15,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:16,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:16,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:17,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:17,782 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:09:18,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:19,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:09:19,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:09:19,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:20,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:21,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:22,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:22,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:23,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:24,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:25,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:26,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:27,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:27,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:27,939 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:09:28,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:29,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:30,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:31,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:32,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:33,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:33,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:34,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:09:34,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:09:34,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:35,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:36,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:37,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:38,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:38,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:39,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:40,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:41,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:42,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:43,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:44,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:44,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:09:44,590 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:09:44,591 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:09:44,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:44,592 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:09:44,956 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:09:45,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:45,957 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:09:46,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:47,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:47,786 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:09:47,958 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:09:48,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:49,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:09:49,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:09:49,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:50,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:50,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:51,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:52,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:53,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:54,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:55,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:55,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:09:56,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:57,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:58,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:59,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:09:59,964 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:10:00,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:00,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:01,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:02,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:03,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:04,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:10:04,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:10:04,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:05,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:06,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:06,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:07,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:08,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:09,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:10,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:11,400 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:11,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:12,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:13,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:14,404 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:15,405 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:15,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:10:15,599 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:10:15,599 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:10:15,601 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:10:15,985 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:10:16,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:17,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:17,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:17,790 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:10:17,987 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:10:18,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:19,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:10:19,025 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:10:19,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:20,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:21,407 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:22,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:22,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:23,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:24,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:25,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:26,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:27,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:27,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:28,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:29,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:30,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:31,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:31,996 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:10:32,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:32,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:33,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:34,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:10:34,026 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:10:34,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:35,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:36,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:37,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:37,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:38,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:39,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:40,409 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:41,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:42,410 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:43,418 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:43,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:44,412 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:45,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:46,416 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:47,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:47,800 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:10:48,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:48,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:49,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:10:49,026 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:10:49,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:10:49,158 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:10:49,158 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:10:49,160 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:10:49,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:50,018 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:10:50,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:51,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:52,020 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:10:52,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:53,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:54,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:54,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:10:55,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:56,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:57,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:58,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:10:59,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:00,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:00,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:01,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:02,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:03,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:04,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:11:04,026 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:11:04,032 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:11:04,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:05,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:05,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:06,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:07,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:08,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:09,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:10,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:10,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:11,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:12,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:13,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:14,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:15,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:15,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:16,575 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:17,383 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:11:17,385 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:11:17,385 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:11:17,387 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:11:17,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:17,802 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:11:18,053 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:11:18,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:19,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:11:19,026 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:11:19,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:20,055 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:11:20,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:21,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:21,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:22,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:23,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:24,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:25,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:26,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:27,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:27,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:28,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:29,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:30,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:31,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:32,063 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:11:32,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:32,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:33,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:34,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:11:34,026 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:11:34,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:35,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:36,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:37,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:38,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:38,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:39,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:40,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:41,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:42,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:43,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:43,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:44,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:45,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:46,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:47,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:47,809 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:11:48,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:48,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:48,966 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:11:48,968 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:11:48,968 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:11:48,970 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:11:49,089 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:11:49,375 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:11:49,375 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:11:49,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:50,090 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:11:50,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:51,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:52,091 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:11:52,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:53,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:54,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:11:54,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:55,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:56,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:57,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:58,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:11:59,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:00,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:00,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:01,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:02,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:03,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:04,099 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:12:04,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:12:04,109 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:12:04,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:05,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:05,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:06,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:07,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:08,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:09,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:10,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:10,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:11,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:12,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:13,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:14,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:15,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:15,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:16,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:17,810 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:12:17,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:18,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:19,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:12:19,109 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:12:19,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:12:19,674 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:12:19,674 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:12:19,679 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:12:20,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:20,121 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:12:21,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:21,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:22,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:22,123 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:12:23,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:24,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:25,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:26,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:27,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:27,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:28,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:29,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:30,076 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:31,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:32,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:33,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:33,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:34,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:12:34,110 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:12:34,132 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:12:34,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:35,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:36,159 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:37,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:38,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:38,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:39,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:40,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:41,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:42,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:43,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:43,779 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:44,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:45,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:46,165 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:47,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:47,814 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:12:48,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:48,424 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:12:48,426 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:12:48,427 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:12:48,428 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:12:48,845 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:49,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:12:49,110 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:12:49,147 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:12:49,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:50,148 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:12:50,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:51,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:52,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:53,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:54,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:54,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:12:55,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:56,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:57,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:58,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:59,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:12:59,979 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:00,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:01,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:02,155 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:13:02,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:03,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:04,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:13:04,110 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:13:04,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:05,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:05,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:06,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:07,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:08,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:09,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:10,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:10,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:11,168 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:12,175 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:13,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:14,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:15,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:15,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:16,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:17,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:17,413 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:13:17,415 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:13:17,415 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:13:17,416 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:13:17,818 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:13:18,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:18,186 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:13:19,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:13:19,111 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:13:19,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:20,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:20,188 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:13:20,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:21,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:22,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:23,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:24,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:25,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:26,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:26,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:27,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:28,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:29,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:30,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:31,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:31,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:32,195 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:13:32,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:33,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:34,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:13:34,110 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:13:34,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:35,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:36,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:37,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:37,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:38,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:39,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:40,624 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:41,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:42,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:42,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:43,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:44,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:45,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:46,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:47,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:13:47,389 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:13:47,390 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:13:47,391 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:13:47,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:47,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:47,822 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:13:48,218 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:13:48,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:49,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:13:49,110 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:13:49,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:50,220 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:13:50,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:51,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:52,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:53,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:53,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:54,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:55,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:56,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:57,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:13:58,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:13:58,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:00,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:01,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:02,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:02,227 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:14:03,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:03,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:04,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:04,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:14:04,110 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:14:05,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:06,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:07,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:08,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:09,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:09,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:10,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:11,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:12,022 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:13,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:14,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:14,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:15,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:16,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:17,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:17,829 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:14:18,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:18,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:14:18,295 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:14:18,296 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:14:18,297 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:14:19,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:19,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:14:19,111 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:14:19,244 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:14:20,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:20,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:20,245 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:14:21,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:22,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:23,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:24,029 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:25,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:25,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:26,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:27,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:28,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:29,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:30,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:30,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:31,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:32,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:33,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:34,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:34,110 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:14:34,111 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:14:34,253 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:14:35,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:35,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:36,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:37,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:38,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:39,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:40,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:40,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:41,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:42,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:43,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:44,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:45,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:45,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:46,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:47,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:47,829 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:14:48,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:49,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:49,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:14:49,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:14:49,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:14:49,761 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:14:49,761 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:14:49,763 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:14:50,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:50,273 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:14:51,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:51,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:52,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:52,274 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:14:53,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:54,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:55,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:56,191 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:57,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:57,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:14:58,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:14:59,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:00,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:01,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:02,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:02,280 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:15:02,834 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:03,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:04,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:15:04,114 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:15:04,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:05,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:06,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:07,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:08,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:08,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:09,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:10,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:11,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:12,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:13,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:13,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:14,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:15,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:16,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:17,002 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:15:17,004 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:15:17,005 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:15:17,007 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:15:17,298 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:15:17,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:17,833 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:15:18,299 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:15:18,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:18,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:19,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:15:19,114 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:15:19,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:20,301 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:15:20,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:21,414 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:22,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:23,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:24,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:24,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:25,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:26,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:27,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:28,415 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:29,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:30,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:30,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:31,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:32,308 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:15:32,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:33,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:34,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:15:34,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:15:34,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:35,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:35,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:36,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:37,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:38,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:39,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:40,526 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:40,796 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:41,540 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:42,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:43,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:44,535 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:45,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:45,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:46,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:47,068 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:15:47,069 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:15:47,069 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:15:47,074 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:15:47,327 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:15:47,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:47,837 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:15:48,327 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:15:48,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:49,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:15:49,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:15:49,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:50,328 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:15:50,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:50,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:51,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:52,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:53,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:54,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:55,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:55,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:15:56,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:57,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:58,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:15:59,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:00,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:00,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:01,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:02,335 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:16:02,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:03,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:04,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:16:04,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:16:04,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:05,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:06,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:06,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:07,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:08,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:09,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:10,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:11,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:11,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:12,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:13,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:14,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:15,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:16,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:16,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:17,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:17,847 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:16:18,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:19,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:16:19,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:16:19,161 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:16:19,180 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:16:19,180 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:16:19,182 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:16:19,357 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:16:19,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:20,358 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:16:20,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:21,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:22,360 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:16:22,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:22,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:23,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:24,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:25,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:26,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:27,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:27,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:28,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:29,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:30,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:31,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:32,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:33,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:33,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:34,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:16:34,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:16:34,368 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:16:34,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:35,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:36,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:37,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:38,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:38,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:39,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:40,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:41,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:42,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:43,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:43,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:44,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:45,582 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:46,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:47,583 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:47,850 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:16:48,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:48,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:49,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:16:49,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:16:49,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:16:49,338 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:16:49,338 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:16:49,339 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:16:49,382 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:16:49,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:50,383 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:16:50,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:51,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:52,385 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:16:52,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:53,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:54,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:54,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:16:55,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:56,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:57,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:58,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:16:59,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:00,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:00,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:01,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:02,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:03,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:04,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:17:04,115 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:17:04,393 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:17:04,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:05,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:05,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:06,745 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:07,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:08,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:09,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:10,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:10,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:11,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:12,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:13,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:14,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:15,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:15,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:16,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:17,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:17,854 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:17:18,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:19,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:17:19,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:17:19,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:19,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:17:19,875 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:17:19,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:17:19,876 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:17:20,415 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:17:20,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:21,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:21,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:22,416 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:17:22,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:23,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:24,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:25,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:26,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:27,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:27,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:28,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:29,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:30,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:31,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:32,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:33,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:33,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:34,115 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:17:34,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:17:34,423 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:17:34,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:35,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:36,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:37,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:38,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:38,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:39,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:40,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:41,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:42,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:43,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:43,816 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:44,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:45,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:46,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:47,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:47,862 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:17:48,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:48,863 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:49,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:17:49,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:17:49,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:17:49,645 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:17:49,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:17:49,647 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:17:50,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:50,445 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:17:50,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:51,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:52,453 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:17:52,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:53,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:54,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:54,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:17:55,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:56,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:57,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:58,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:59,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:17:59,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:00,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:01,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:02,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:03,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:04,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:18:04,116 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:18:04,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:05,521 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:05,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:06,463 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:18:06,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:07,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:08,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:09,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:10,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:10,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:11,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:12,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:13,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:14,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:15,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:15,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:16,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:17,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:17,860 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:18:18,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:19,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:18:19,126 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:18:19,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:20,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:20,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:21,797 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:22,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:23,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:24,357 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:18:24,358 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:18:24,359 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:18:24,363 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:18:24,483 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:18:24,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:25,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:25,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:26,484 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:18:26,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:27,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:28,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:29,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:30,800 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:30,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:31,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:32,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:33,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:34,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:18:34,123 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:18:34,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:35,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:36,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:36,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:37,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:38,491 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:18:38,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:39,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:40,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:41,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:41,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:42,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:43,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:44,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:45,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:46,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:47,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:47,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:47,865 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:18:48,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:49,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:18:49,141 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:18:49,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:50,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:51,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:52,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:52,820 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:53,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:18:53,620 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:18:53,620 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:18:53,621 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:18:54,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:54,513 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:18:55,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:56,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:56,514 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:18:57,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:58,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:18:58,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:18:59,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:00,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:01,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:02,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:03,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:03,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:04,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:04,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:19:04,128 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:19:05,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:06,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:07,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:08,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:08,522 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:19:09,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:09,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:10,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:11,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:12,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:13,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:14,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:14,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:15,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:16,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:17,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:17,868 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:19:18,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:19,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:19:19,142 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:19:19,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:20,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:20,250 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:21,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:22,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:23,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:24,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:19:24,168 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:19:24,169 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:19:24,170 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:19:24,544 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:19:24,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:25,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:25,581 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:26,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:26,546 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:19:27,308 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:28,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:29,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:30,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:30,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:31,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:32,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:33,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:34,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:19:34,131 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:19:34,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:35,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:36,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:36,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:37,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:38,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:39,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:40,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:40,556 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:19:41,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:41,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:42,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:43,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:44,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:45,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:46,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:47,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:47,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:47,871 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:19:48,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:49,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:19:49,131 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:19:49,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:50,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:51,311 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:52,321 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:53,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:53,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:54,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:55,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:56,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:57,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:58,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:19:58,229 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:19:58,230 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:19:58,234 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:19:58,578 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:19:59,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:19:59,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:19:59,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:00,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:00,580 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:20:01,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:02,378 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:03,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:04,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:20:04,132 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:20:04,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:04,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:05,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:06,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:07,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:08,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:09,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:09,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:10,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:11,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:12,379 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:13,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:14,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:14,589 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:20:15,278 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:15,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:16,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:17,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:17,875 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:20:18,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:19,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:20:19,132 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:20:19,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:20,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:20,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:21,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:22,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:23,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:24,380 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:25,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:25,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:26,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:27,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:28,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:29,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:30,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:30,827 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:31,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:20:31,195 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:20:31,195 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:20:31,200 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:20:31,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:31,609 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:20:32,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:32,610 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:20:33,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:34,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:20:34,132 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:20:34,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:35,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:35,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:36,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:37,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:38,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:39,605 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:40,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:41,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:41,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:42,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:43,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:44,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:45,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:46,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:46,619 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:20:46,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:47,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:47,879 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:20:48,606 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:49,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:20:49,132 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:20:49,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:50,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:51,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:52,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:52,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:53,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:54,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:55,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:56,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:57,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:57,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:20:58,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:20:59,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:00,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:01,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:01,985 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:21:01,987 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:21:01,987 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:21:01,990 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:21:02,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:02,636 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:21:03,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:03,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:04,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:21:04,132 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:21:04,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:04,637 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:21:05,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:06,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:07,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:08,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:08,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:09,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:10,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:11,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:12,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:13,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:14,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:14,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:15,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:16,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:16,645 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:21:17,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:17,883 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:21:18,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:19,132 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:21:19,132 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:21:19,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:20,298 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:20,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:21,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:22,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:23,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:24,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:25,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:25,831 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:26,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:27,619 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:28,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:29,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:30,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:30,839 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:31,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:32,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:33,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:33,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:21:33,836 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:21:33,837 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:21:33,842 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:21:34,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:21:34,249 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:21:34,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:34,666 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:21:35,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:36,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:36,667 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:21:36,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:37,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:38,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:39,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:40,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:41,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:42,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:42,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:43,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:44,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:45,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:46,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:47,627 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:47,888 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:21:47,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:48,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:48,675 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:21:49,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:21:49,248 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:21:49,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:50,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:51,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:52,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:53,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:53,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:54,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:55,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:56,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:57,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:58,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:21:58,841 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:21:59,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:00,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:01,634 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:02,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:03,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:03,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:04,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:22:04,248 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:22:04,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:04,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:22:04,779 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:22:04,780 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:22:04,785 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:22:05,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:05,692 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:22:06,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:06,693 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:22:07,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:08,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:08,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:09,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:10,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:11,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:12,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:13,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:14,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:14,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:15,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:16,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:17,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:17,891 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:22:18,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:18,701 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:22:19,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:22:19,248 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:22:19,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:20,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:20,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:21,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:22,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:23,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:24,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:25,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:25,844 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:26,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:27,640 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:28,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:29,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:30,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:30,848 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:31,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:32,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:33,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:33,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:22:33,886 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:22:33,886 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:22:33,888 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:22:34,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:22:34,343 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:22:34,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:34,722 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:22:34,723 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:22:35,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:36,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:36,724 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:22:36,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:37,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:38,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:39,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:40,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:41,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:41,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:42,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:43,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:44,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:45,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:46,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:47,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:47,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:47,895 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:22:48,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:48,732 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:22:49,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:22:49,341 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:22:49,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:50,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:51,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:52,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:52,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:53,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:54,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:55,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:56,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:57,645 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:58,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:22:58,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:22:59,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:00,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:01,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:02,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:03,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:03,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:04,143 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:23:04,144 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:23:04,145 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:23:04,146 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:23:04,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:23:04,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:23:04,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:04,752 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:23:05,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:06,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:06,753 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:23:07,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:08,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:09,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:09,853 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:10,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:11,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:12,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:13,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:14,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:15,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:15,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:16,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:16,759 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:23:17,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:17,900 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:23:18,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:19,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:23:19,545 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:23:19,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:20,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:20,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:21,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:22,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:23,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:24,655 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:25,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:25,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:26,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:27,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:28,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:29,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:30,661 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:31,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:23:31,223 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:23:31,223 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:23:31,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:31,225 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:23:31,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:31,777 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:23:32,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:32,778 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:23:33,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:34,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:23:34,545 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:23:34,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:35,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:36,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:36,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:37,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:38,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:39,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:40,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:41,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:42,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:42,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:43,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:44,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:44,785 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:23:45,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:46,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:47,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:47,904 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:23:47,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:48,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:49,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:23:49,545 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:23:49,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:50,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:51,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:52,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:53,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:53,857 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:54,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:55,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:56,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:57,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:58,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:23:58,865 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:23:59,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:00,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:01,351 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:24:01,352 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:24:01,352 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:24:01,353 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:24:01,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:01,803 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:24:02,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:02,804 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:24:03,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:04,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:24:04,545 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:24:04,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:04,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:05,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:06,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:07,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:08,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:09,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:09,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:10,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:11,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:12,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:13,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:14,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:15,309 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:15,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:16,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:16,813 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:24:17,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:17,908 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:24:18,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:19,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:24:19,545 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:24:19,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:20,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:20,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:21,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:22,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:23,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:24,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:25,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:25,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:26,770 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:27,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:28,764 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:29,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:30,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:30,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:31,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:32,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:32,801 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:24:32,803 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:24:32,803 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:24:32,805 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:24:32,833 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:24:33,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:34,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:24:34,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:24:34,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:34,835 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:24:35,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:36,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:36,861 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:37,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:38,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:39,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:40,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:41,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:42,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:42,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:43,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:44,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:45,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:46,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:46,844 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:24:47,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:47,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:47,910 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:24:48,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:49,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:24:49,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:24:49,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:50,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:51,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:52,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:53,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:53,862 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:54,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:55,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:56,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:57,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:58,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:24:58,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:24:59,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:00,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:01,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:02,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:25:02,035 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:25:02,036 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:25:02,037 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:25:02,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:02,859 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:25:02,860 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:25:03,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:04,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:04,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:25:04,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:25:04,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:04,861 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:25:05,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:06,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:07,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:08,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:09,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:09,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:10,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:11,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:12,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:13,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:14,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:15,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:15,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:16,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:16,868 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:25:17,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:17,915 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:25:18,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:19,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:25:19,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:25:19,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:20,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:20,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:21,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:22,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:23,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:24,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:25,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:25,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:26,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:27,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:28,791 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:29,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:30,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:30,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:31,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:32,775 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:25:32,776 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:25:32,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:25:32,779 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:25:32,892 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:25:33,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:34,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:34,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:25:34,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:25:34,893 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:25:35,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:36,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:36,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:37,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:38,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:39,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:40,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:41,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:42,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:42,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:43,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:44,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:45,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:46,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:46,901 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:25:47,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:47,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:47,917 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:25:48,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:49,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:49,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:25:49,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:25:50,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:51,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:52,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:53,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:53,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:54,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:55,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:56,190 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:57,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:58,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:25:58,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:25:59,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:00,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:01,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:02,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:02,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:26:02,777 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:26:02,777 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:26:02,778 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:26:02,921 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:26:03,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:04,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:04,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:04,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:26:04,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:26:04,923 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:26:05,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:06,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:07,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:08,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:09,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:09,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:10,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:11,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:12,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:13,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:14,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:14,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:15,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:16,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:16,930 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:26:17,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:17,922 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:26:18,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:19,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:19,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:26:19,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:26:20,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:20,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:21,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:22,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:23,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:24,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:25,317 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:25,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:26,319 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:27,329 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:28,322 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:29,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:30,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:30,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:31,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:32,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:32,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:26:32,698 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:26:32,699 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:26:32,700 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:26:32,949 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:26:33,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:34,325 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:34,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:26:34,546 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:26:34,951 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:26:35,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:35,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:36,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:37,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:38,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:39,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:40,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:40,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:41,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:42,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:43,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:44,326 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:45,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:45,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:46,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:46,959 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:26:47,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:47,926 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:26:48,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:49,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:49,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:26:49,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:26:50,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:51,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:51,879 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:52,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:53,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:54,327 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:55,328 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:56,332 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:56,898 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:26:57,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:58,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:26:59,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:00,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:01,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:01,543 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:27:01,544 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:27:01,545 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:27:01,547 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:27:01,957 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:01,977 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:27:02,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:02,978 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:27:03,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:04,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:04,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:27:04,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:27:05,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:06,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:07,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:07,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:08,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:09,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:10,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:11,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:12,341 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:13,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:13,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:14,370 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:14,984 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:27:15,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:16,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:17,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:17,930 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:27:18,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:18,932 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:19,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:19,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:27:19,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:27:20,342 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:21,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:22,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:23,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:24,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:24,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:25,346 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:26,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:27,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:28,348 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:29,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:30,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:30,598 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:27:30,598 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:27:30,599 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:27:30,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:30,600 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:27:31,010 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:27:31,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:32,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:33,011 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:27:33,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:34,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:34,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:27:34,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:27:35,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:35,642 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:36,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:37,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:38,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:39,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:40,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:41,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:41,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:42,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:43,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:44,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:45,018 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:27:45,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:46,004 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:46,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:47,353 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:47,933 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:27:48,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:49,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:49,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:27:49,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:27:50,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:51,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:52,003 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:52,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:53,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:54,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:55,365 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:56,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:57,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:27:57,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:58,362 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:27:59,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:00,361 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:01,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:28:01,068 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:28:01,069 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:28:01,070 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:28:01,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:02,049 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:28:02,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:02,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:03,050 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:28:03,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:04,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:04,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:28:04,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:28:05,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:06,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:07,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:08,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:08,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:09,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:10,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:11,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:12,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:13,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:14,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:14,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:15,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:16,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:17,059 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:28:17,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:17,938 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:28:18,479 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:19,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:19,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:28:19,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:28:19,660 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:20,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:21,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:22,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:23,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:24,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:25,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:25,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:26,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:27,486 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:28,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:29,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:30,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:30,485 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:31,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:32,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:33,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:28:33,143 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:28:33,143 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:28:33,145 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:28:33,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:34,087 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:28:34,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:28:34,547 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:28:34,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:35,088 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:28:35,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:35,717 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:36,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:37,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:38,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:39,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:40,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:41,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:41,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:42,552 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:43,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:44,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:45,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:46,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:47,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:47,096 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:28:47,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:47,942 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:28:48,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:49,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:28:49,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:28:49,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:50,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:51,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:52,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:53,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:53,553 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:54,554 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:55,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:56,560 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:57,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:58,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:28:58,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:28:59,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:00,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:01,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:01,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:29:01,637 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:29:01,637 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:29:01,639 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:29:02,123 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:29:02,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:03,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:03,124 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:29:03,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:04,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:29:04,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:29:04,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:05,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:06,563 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:07,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:08,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:09,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:09,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:10,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:11,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:12,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:13,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:14,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:14,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:15,131 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:29:15,564 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:16,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:17,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:17,946 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:29:18,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:19,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:29:19,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:29:19,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:19,641 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:20,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:21,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:22,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:23,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:24,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:25,017 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:25,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:26,565 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:27,571 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:28,566 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:29,572 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:30,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:30,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:31,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:32,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:29:32,017 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:29:32,017 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:29:32,019 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:29:32,156 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:29:32,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:33,157 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:29:33,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:34,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:29:34,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:29:34,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:35,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:35,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:36,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:37,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:38,567 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:39,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:40,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:41,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:41,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:42,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:43,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:44,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:45,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:46,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:46,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:47,167 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:29:47,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:47,950 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:29:48,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:49,547 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:29:49,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:29:49,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:50,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:51,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:52,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:52,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:53,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:54,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:55,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:56,570 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:57,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:29:57,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:58,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:29:59,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:00,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:01,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:02,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:02,577 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:03,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:30:03,016 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:30:03,017 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:30:03,018 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:30:03,189 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:30:03,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:04,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:30:04,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:30:04,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:05,192 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:30:05,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:06,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:07,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:08,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:08,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:09,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:10,578 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:11,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:12,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:13,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:14,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:14,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:15,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:16,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:17,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:17,954 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:30:18,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:19,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:19,204 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:30:19,548 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:30:19,548 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:30:19,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:20,579 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:21,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:22,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:23,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:24,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:25,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:25,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:26,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:27,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:28,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:29,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:30,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:30,580 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:31,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:32,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:33,586 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:34,555 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:30:34,556 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:30:34,599 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:35,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:35,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:36,587 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:37,221 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:30:37,222 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:30:37,222 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:30:37,223 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:30:37,227 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:30:37,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:38,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:39,229 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:30:39,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:40,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:41,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:41,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:42,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:43,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:44,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:45,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:46,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:46,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:47,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:47,958 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:30:48,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:49,686 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:30:49,686 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:30:49,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:50,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:51,237 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:30:51,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:52,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:52,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:53,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:54,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:55,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:56,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:57,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:30:57,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:58,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:30:59,685 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:00,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:01,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:02,033 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:02,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:03,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:04,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:31:04,603 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:31:04,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:05,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:06,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:31:06,293 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:31:06,294 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:31:06,295 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:31:06,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:07,257 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:31:07,258 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:31:07,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:07,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:08,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:09,259 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:31:09,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:10,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:11,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:12,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:13,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:13,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:14,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:15,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:16,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:17,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:17,962 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:31:18,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:18,964 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:19,603 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:31:19,603 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:31:19,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:20,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:21,266 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:31:21,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:22,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:23,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:24,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:24,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:25,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:26,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:27,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:28,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:29,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:30,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:30,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:31,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:32,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:33,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:34,611 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:31:34,612 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:31:34,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:35,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:35,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:36,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:37,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:38,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:31:38,667 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:31:38,667 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:31:38,668 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:31:39,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:39,288 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:31:39,288 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:31:40,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:41,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:41,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:41,289 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:31:42,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:43,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:44,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:45,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:46,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:47,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:47,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:47,966 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:31:48,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:49,109 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:49,608 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:31:49,608 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:31:50,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:51,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:51,296 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:31:52,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:52,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:53,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:54,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:55,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:56,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:57,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:58,031 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:31:58,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:31:59,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:00,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:01,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:02,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:03,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:03,120 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:04,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:04,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:32:04,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:32:05,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:06,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:07,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:07,569 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:32:07,571 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:32:07,572 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:32:07,574 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:32:08,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:08,316 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:32:08,980 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:09,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:09,317 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:32:10,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:11,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:12,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:13,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:14,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:14,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:15,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:16,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:17,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:17,971 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:32:18,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:19,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:19,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:32:19,612 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:32:19,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:20,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:21,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:22,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:23,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:23,325 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:32:24,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:24,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:25,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:26,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:27,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:28,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:29,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:30,035 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:30,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:31,130 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:32,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:33,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:34,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:34,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:32:34,612 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:32:35,131 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:35,694 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:36,135 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:37,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:38,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:39,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:40,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:41,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:41,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:41,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:32:41,750 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:32:41,750 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:32:41,752 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:32:42,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:42,342 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:32:43,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:43,343 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:32:44,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:45,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:46,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:47,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:47,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:47,974 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:32:48,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:49,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:49,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:32:49,612 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:32:50,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:51,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:52,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:53,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:53,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:54,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:55,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:56,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:57,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:57,351 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:32:58,095 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:32:58,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:32:59,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:00,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:01,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:02,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:03,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:04,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:04,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:04,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:33:04,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:33:05,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:06,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:07,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:08,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:09,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:09,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:10,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:11,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:12,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:13,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:13,500 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:33:13,502 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:33:13,502 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:33:13,503 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:33:14,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:14,366 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:33:14,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:15,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:15,367 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:33:16,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:17,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:17,978 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:33:18,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:19,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:19,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:33:19,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:33:20,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:20,738 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:21,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:22,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:23,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:24,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:25,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:26,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:26,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:27,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:28,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:29,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:29,375 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:33:30,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:31,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:31,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:32,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:33,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:34,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:34,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:33:34,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:33:35,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:36,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:37,057 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:37,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:38,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:39,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:40,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:41,211 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:42,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:42,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:43,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:44,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:44,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:33:44,828 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:33:44,829 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:33:44,830 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:33:45,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:45,392 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:33:45,393 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:33:46,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:47,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:47,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:47,393 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:33:47,982 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:33:48,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:49,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:49,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:33:49,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:33:50,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:51,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:52,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:53,059 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:53,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:54,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:55,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:56,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:57,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:58,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:33:59,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:33:59,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:00,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:01,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:01,401 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:34:02,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:03,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:04,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:04,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:04,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:34:04,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:34:05,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:06,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:07,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:08,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:09,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:10,060 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:10,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:11,282 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:12,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:13,288 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:14,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:15,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:15,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:16,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:17,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:17,493 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:34:17,495 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:34:17,495 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:34:17,496 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:34:17,987 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:34:18,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:18,423 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:34:19,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:19,424 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:34:19,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:34:19,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:34:20,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:20,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:21,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:22,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:23,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:24,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:25,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:26,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:26,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:27,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:28,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:29,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:30,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:31,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:31,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:31,431 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:34:32,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:33,292 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:34,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:34,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:34:34,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:34:35,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:36,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:37,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:37,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:38,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:39,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:40,305 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:41,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:42,073 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:42,299 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:43,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:44,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:45,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:45,931 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:34:45,932 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:34:45,933 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:34:45,935 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:34:46,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:46,456 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:34:47,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:47,340 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:47,457 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:34:47,991 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:34:48,336 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:49,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:49,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:34:49,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:34:50,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:51,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:52,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:53,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:53,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:54,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:55,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:56,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:57,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:58,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:58,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:34:59,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:34:59,464 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:35:00,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:01,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:02,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:03,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:04,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:04,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:04,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:35:04,613 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:35:05,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:06,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:07,338 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:08,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:09,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:10,075 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:10,339 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:11,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:12,343 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:13,354 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:14,352 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:15,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:15,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:16,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:16,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:35:16,823 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:35:16,824 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:35:16,825 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:35:17,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:17,484 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:35:17,485 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:35:17,995 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:35:18,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:19,349 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:19,486 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:35:19,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:35:19,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:35:20,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:20,729 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:21,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:22,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:23,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:24,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:25,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:26,079 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:26,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:27,350 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:28,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:29,492 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:35:29,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:30,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:31,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:31,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:32,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:33,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:34,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:35:34,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:35:34,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:35,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:36,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:36,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:37,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:38,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:39,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:40,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:41,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:41,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:42,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:43,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:44,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:45,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:46,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:35:46,147 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:35:46,147 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:35:46,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:46,149 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:35:46,511 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:35:46,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:47,512 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:35:47,635 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:47,999 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:35:48,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:49,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:35:49,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:35:49,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:50,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:51,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:52,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:52,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:53,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:54,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:55,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:56,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:57,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:35:57,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:58,636 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:35:59,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:00,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:01,521 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:36:01,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:02,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:02,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:03,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:04,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:36:04,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:36:04,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:05,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:06,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:07,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:07,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:08,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:09,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:10,638 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:11,644 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:12,092 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:12,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:13,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:14,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:15,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:16,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:16,787 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:36:16,789 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:36:16,789 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:36:16,791 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:36:17,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:17,542 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:36:17,542 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:36:17,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:18,003 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:36:18,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:19,543 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:36:19,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:36:19,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:36:19,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:20,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:21,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:22,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:23,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:23,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:24,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:25,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:26,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:27,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:28,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:29,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:29,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:30,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:31,551 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:36:31,651 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:32,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:33,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:34,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:34,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:36:34,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:36:34,654 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:35,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:36,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:37,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:38,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:39,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:40,085 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:40,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:41,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:42,652 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:43,668 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:44,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:45,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:45,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:46,658 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:47,659 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:48,009 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:36:48,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:49,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:36:49,147 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:36:49,147 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:36:49,148 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:36:49,574 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:36:49,575 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:36:49,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:36:49,614 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:36:49,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:50,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:50,776 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:51,575 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:36:51,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:52,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:53,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:54,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:55,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:56,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:36:56,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:57,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:58,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:36:59,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:00,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:01,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:01,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:02,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:03,584 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:37:03,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:04,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:37:04,615 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:37:04,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:05,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:06,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:07,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:07,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:08,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:09,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:10,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:11,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:12,665 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:13,088 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:13,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:14,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:15,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:16,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:17,669 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:18,014 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:37:18,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:18,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:37:18,978 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:37:18,978 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:37:18,978 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:18,979 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:37:19,613 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:37:19,614 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:37:19,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:37:19,615 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:37:19,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:20,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:21,615 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:37:21,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:22,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:23,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:24,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:24,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:25,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:26,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:27,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:28,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:29,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:30,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:30,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:31,670 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:32,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:33,622 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:37:33,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:34,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:37:34,615 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:37:34,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:35,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:35,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:36,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:37,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:38,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:39,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:40,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:41,090 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:41,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:42,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:43,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:44,672 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:45,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:46,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:46,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:47,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:48,021 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:37:48,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:49,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:37:49,616 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:37:49,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:50,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:50,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:37:50,742 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:37:50,742 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:37:50,743 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:37:51,157 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:51,647 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:37:51,648 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:37:51,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:52,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:53,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:54,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:55,676 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:56,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:57,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:37:57,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:58,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:37:59,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:00,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:01,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:02,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:03,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:03,677 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:04,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:38:04,616 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:38:04,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:05,656 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:38:05,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:06,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:07,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:08,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:08,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:09,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:10,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:11,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:12,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:13,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:14,157 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:14,678 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:15,679 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:16,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:17,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:18,026 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:38:18,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:19,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:38:19,622 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:38:19,680 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:19,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:20,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:21,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:21,915 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:38:21,916 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:38:21,916 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:38:21,918 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:38:22,676 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:38:22,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:23,677 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:38:23,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:24,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:25,316 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:25,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:26,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:27,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:28,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:29,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:30,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:31,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:31,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:32,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:33,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:34,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:38:34,616 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:38:34,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:35,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:35,685 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:38:36,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:37,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:37,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:38,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:39,683 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:40,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:41,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:42,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:42,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:43,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:44,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:45,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:46,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:47,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:47,684 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:48,030 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:38:48,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:49,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:38:49,616 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:38:49,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:50,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:51,693 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:52,172 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:52,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:53,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:38:53,311 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:38:53,311 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:38:53,312 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:38:53,703 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:38:53,704 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:38:53,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:54,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:55,706 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:38:55,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:56,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:57,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:58,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:38:58,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:38:59,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:00,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:01,723 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:02,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:03,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:03,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:04,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:39:04,617 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:39:04,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:05,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:06,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:07,713 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:39:07,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:08,460 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:08,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:09,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:10,724 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:11,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:12,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:13,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:14,170 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:14,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:15,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:16,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:17,725 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:18,028 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:39:18,726 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:19,628 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:39:19,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:39:19,731 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:19,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:20,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:21,735 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:22,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:23,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:24,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:24,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:39:24,790 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:39:24,790 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:39:24,790 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:24,795 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:39:25,734 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:39:25,734 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:39:25,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:26,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:27,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:28,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:29,739 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:30,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:30,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:31,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:32,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:33,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:34,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:39:34,621 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:39:34,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:35,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:35,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:36,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:37,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:38,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:39,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:39,744 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:39:40,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:41,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:41,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:42,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:43,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:44,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:45,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:46,173 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:46,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:47,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:48,032 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:39:48,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:49,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:39:49,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:39:49,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:50,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:51,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:51,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:52,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:53,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:54,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:54,989 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:39:54,990 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:39:54,991 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:39:54,994 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:39:55,746 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:55,759 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:39:55,759 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:39:56,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:57,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:39:57,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:58,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:39:59,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:00,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:01,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:02,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:03,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:03,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:04,625 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:40:04,625 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:40:04,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:05,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:06,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:07,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:07,766 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:40:08,360 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:08,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:09,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:10,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:11,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:12,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:13,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:14,177 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:14,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:15,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:16,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:17,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:18,037 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:40:18,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:19,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:40:19,637 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:40:19,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:19,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:20,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:21,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:22,754 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:23,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:24,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:24,892 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:40:24,894 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:40:24,894 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:40:24,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:24,896 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:40:25,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:25,787 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:40:25,788 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:40:26,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:27,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:28,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:29,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:30,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:30,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:31,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:32,755 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:33,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:34,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:40:34,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:40:34,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:35,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:35,777 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:36,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:37,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:37,797 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:40:38,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:39,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:40,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:41,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:41,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:42,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:43,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:44,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:45,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:46,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:46,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:47,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:48,040 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:40:48,757 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:49,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:40:49,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:40:49,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:50,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:51,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:51,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:52,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:53,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:54,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:54,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:40:54,877 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:40:54,878 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:40:54,879 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:40:55,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:55,819 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:40:55,820 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:40:56,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:40:56,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:57,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:58,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:40:59,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:00,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:01,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:02,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:02,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:03,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:04,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:41:04,629 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:41:04,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:05,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:06,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:07,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:07,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:08,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:09,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:09,827 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:41:10,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:11,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:12,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:12,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:13,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:14,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:15,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:16,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:17,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:18,044 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:41:18,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:18,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:19,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:41:19,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:41:19,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:20,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:21,786 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:22,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:23,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:23,773 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:24,772 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:25,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:26,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:27,250 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:41:27,251 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:41:27,252 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:41:27,257 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:41:27,774 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:27,847 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:41:27,847 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:41:28,663 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:28,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:29,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:29,848 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:41:30,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:31,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:32,780 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:33,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:34,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:34,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:41:34,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:41:34,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:35,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:36,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:37,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:38,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:39,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:39,854 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:41:40,084 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:40,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:41,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:42,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:43,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:44,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:45,187 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:45,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:46,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:47,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:48,048 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:41:48,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:49,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:41:49,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:41:49,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:50,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:50,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:51,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:52,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:53,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:54,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:55,793 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:56,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:41:56,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:41:56,614 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:41:56,615 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:41:56,616 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:41:56,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:41:57,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:57,882 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:41:58,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:41:59,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:00,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:01,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:01,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:02,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:03,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:04,012 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:04,629 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:42:04,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:42:05,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:06,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:07,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:07,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:08,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:09,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:09,889 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:42:10,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:11,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:12,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:12,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:13,013 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:14,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:15,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:16,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:17,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:17,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:18,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:18,052 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:42:19,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:19,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:42:19,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:42:20,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:21,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:22,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:22,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:23,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:24,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:25,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:26,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:26,458 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:42:26,460 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:42:26,460 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:42:26,461 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:42:26,910 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:42:27,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:27,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:27,911 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:42:28,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:29,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:30,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:31,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:32,018 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:33,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:33,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:34,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:34,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:42:34,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:42:35,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:36,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:37,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:38,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:39,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:39,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:40,019 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:41,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:41,919 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:42:42,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:43,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:44,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:44,194 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:45,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:46,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:47,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:48,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:48,057 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:42:49,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:49,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:42:49,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:42:49,792 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:50,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:51,020 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:52,027 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:53,024 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:54,034 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:55,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:55,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:42:56,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:57,030 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:57,854 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:42:57,855 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:42:57,855 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:42:57,856 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:42:57,942 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:42:58,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:59,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:42:59,944 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:43:00,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:00,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:01,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:02,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:03,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:04,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:04,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:43:04,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:43:05,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:05,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:06,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:07,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:08,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:09,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:10,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:11,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:11,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:11,952 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:43:12,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:13,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:14,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:15,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:16,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:16,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:17,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:18,060 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:43:18,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:19,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:19,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:43:19,630 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:43:20,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:21,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:22,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:22,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:23,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:24,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:25,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:26,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:27,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:27,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:28,264 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:28,506 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:43:28,508 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:43:28,508 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:43:28,510 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:43:28,971 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:43:29,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:29,972 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:43:30,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:31,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:32,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:33,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:33,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:34,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:34,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:43:34,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:43:35,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:36,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:37,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:38,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:39,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:39,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:40,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:41,591 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:42,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:43,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:44,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:45,001 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:45,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:45,982 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:43:46,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:47,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:48,064 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:43:48,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:49,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:49,630 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:43:49,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:43:50,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:50,765 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:51,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:52,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:53,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:54,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:55,602 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:56,204 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:43:56,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:57,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:58,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:43:59,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:00,601 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:01,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:01,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:44:01,501 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:44:01,501 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:44:01,504 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:44:01,910 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:02,000 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:44:02,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:03,001 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:44:03,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:04,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:44:04,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:44:04,671 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:05,002 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:44:05,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:06,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:06,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:07,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:08,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:09,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:10,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:11,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:12,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:12,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:13,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:14,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:15,842 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:16,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:17,009 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:44:17,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:17,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:18,068 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:44:18,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:19,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:44:19,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:44:19,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:20,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:21,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:22,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:22,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:23,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:24,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:25,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:26,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:27,748 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:28,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:28,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:29,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:30,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:31,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:32,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:33,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:33,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:34,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:44:34,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:44:34,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:34,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:44:34,930 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:44:34,931 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:44:34,936 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:44:35,029 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:44:35,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:36,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:37,033 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:44:37,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:38,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:38,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:39,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:40,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:41,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:42,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:43,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:44,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:44,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:45,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:46,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:47,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:48,072 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:44:48,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:49,040 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:44:49,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:44:49,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:44:49,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:49,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:50,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:51,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:52,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:53,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:54,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:55,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:44:55,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:56,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:57,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:58,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:44:59,756 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:00,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:00,761 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:01,763 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:02,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:03,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:03,912 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:45:03,913 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:45:03,914 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:45:03,916 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:45:04,057 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:45:04,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:45:04,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:45:04,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:05,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:05,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:06,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:07,059 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:45:07,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:08,758 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:09,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:10,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:11,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:11,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:12,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:13,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:14,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:15,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:16,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:16,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:17,066 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:45:17,759 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:18,077 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:45:18,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:19,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:45:19,631 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:45:19,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:20,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:21,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:22,215 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:22,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:23,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:24,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:25,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:26,760 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:27,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:27,762 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:28,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:29,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:30,766 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:31,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:32,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:32,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:33,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:45:33,158 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:45:33,159 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:45:33,160 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:45:33,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:34,088 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:45:34,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:45:34,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:45:34,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:35,088 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:45:35,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:36,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:37,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:37,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:38,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:39,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:40,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:41,768 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:42,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:43,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:43,769 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:44,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:45,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:46,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:47,096 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:45:47,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:48,080 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:45:48,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:49,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:49,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:45:49,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:45:49,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:50,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:51,781 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:52,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:53,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:54,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:54,782 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:55,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:56,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:57,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:58,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:45:59,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:45:59,788 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:00,789 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:01,751 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:46:01,752 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:46:01,753 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:46:01,754 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:46:02,116 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:46:02,158 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:02,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:03,117 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:46:03,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:04,631 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:46:04,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:46:04,733 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:04,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:05,119 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:46:05,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:06,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:07,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:08,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:09,895 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:10,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:10,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:11,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:12,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:13,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:14,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:15,126 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:46:15,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:15,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:16,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:17,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:18,083 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:46:18,896 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:19,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:46:19,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:46:19,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:20,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:21,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:21,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:22,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:23,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:24,897 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:25,905 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:26,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:26,900 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:27,911 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:28,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:29,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:30,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:31,720 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:46:31,721 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:46:31,721 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:46:31,722 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:31,723 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:46:32,139 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:32,151 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:46:33,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:33,153 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:46:34,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:34,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:46:34,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:46:35,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:35,155 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:46:36,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:37,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:37,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:38,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:39,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:40,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:41,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:42,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:42,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:43,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:44,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:45,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:46,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:47,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:47,166 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:46:47,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:48,087 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:46:48,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:49,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:49,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:46:49,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:46:50,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:51,137 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:52,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:53,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:53,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:54,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:55,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:56,138 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:57,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:58,142 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:46:58,238 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:46:59,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:00,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:01,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:02,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:03,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:03,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:47:03,154 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:47:03,155 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:47:03,157 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:47:03,192 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:47:03,561 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:04,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:04,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:47:04,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:47:05,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:05,194 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:47:06,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:07,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:08,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:09,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:09,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:10,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:11,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:12,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:13,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:14,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:15,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:15,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:16,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:17,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:18,091 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:47:18,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:19,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:19,632 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:47:19,632 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:47:20,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:20,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:21,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:21,205 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:47:22,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:23,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:24,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:25,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:26,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:26,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:27,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:28,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:29,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:30,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:31,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:32,155 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:32,241 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:33,149 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:34,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:34,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:47:34,640 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:47:35,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:36,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:37,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:37,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:37,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:47:37,484 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:47:37,485 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:47:37,486 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:47:38,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:38,235 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:47:39,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:39,236 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:47:40,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:41,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:42,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:43,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:43,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:44,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:45,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:46,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:47,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:48,096 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:47:48,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:49,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:49,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:49,637 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:47:49,637 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:47:50,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:51,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:51,246 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:47:52,153 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:53,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:54,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:54,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:47:55,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:56,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:57,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:58,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:47:59,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:00,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:00,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:01,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:02,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:03,163 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:04,157 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:04,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:48:04,639 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:48:05,162 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:05,784 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:06,157 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:06,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:48:06,837 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:48:06,838 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:48:06,839 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:48:07,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:07,269 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:48:08,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:09,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:09,271 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:48:10,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:11,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:11,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:12,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:13,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:14,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:15,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:16,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:16,244 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:17,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:18,099 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:48:18,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:19,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:19,639 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:48:19,639 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:48:20,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:21,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:21,279 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:48:21,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:22,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:23,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:24,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:25,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:26,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:26,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:27,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:28,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:29,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:30,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:31,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:31,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:32,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:33,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:34,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:34,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:48:34,645 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:48:35,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:36,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:36,291 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:37,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:48:37,247 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:48:37,247 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:48:37,249 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:48:37,302 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:48:37,662 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:38,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:39,304 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:48:39,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:40,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:41,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:41,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:42,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:43,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:44,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:45,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:46,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:47,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:47,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:48,103 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:48:48,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:49,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:49,643 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:48:49,643 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:48:50,473 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:51,469 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:52,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:52,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:53,314 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:48:53,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:54,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:55,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:56,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:57,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:48:57,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:58,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:48:59,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:00,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:01,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:02,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:03,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:03,484 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:04,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:04,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:49:04,648 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:49:05,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:06,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:07,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:08,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:08,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:08,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:49:08,683 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:49:08,684 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:49:08,685 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:49:09,331 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:49:09,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:10,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:11,333 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:49:11,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:12,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:13,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:14,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:14,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:15,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:16,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:17,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:18,108 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:49:18,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:19,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:19,647 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:49:19,647 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:49:19,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:20,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:21,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:22,476 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:23,341 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:49:23,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:24,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:25,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:25,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:26,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:27,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:28,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:29,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:30,252 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:30,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:31,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:32,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:33,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:34,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:34,656 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:49:34,657 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:49:35,481 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:35,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:36,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:37,482 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:38,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:49:38,100 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:49:38,100 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:49:38,102 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:49:38,358 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:49:38,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:39,360 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:49:39,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:40,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:41,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:41,361 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:49:41,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:42,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:43,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:44,517 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:45,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:46,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:47,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:47,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:48,112 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:49:48,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:49,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:49,653 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:49:49,653 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:49:50,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:51,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:52,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:53,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:53,369 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:49:53,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:54,518 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:55,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:56,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:57,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:58,256 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:49:58,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:49:59,519 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:00,531 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:01,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:02,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:03,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:03,525 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:04,530 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:04,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:50:04,657 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:50:05,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:06,091 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:50:06,093 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:50:06,093 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:50:06,094 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:50:06,387 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:50:06,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:07,388 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:50:07,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:08,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:08,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:09,389 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:50:09,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:10,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:11,527 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:12,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:13,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:14,503 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:14,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:15,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:16,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:17,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:18,116 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:50:18,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:19,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:19,657 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:50:19,657 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:50:19,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:20,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:21,397 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:50:21,528 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:22,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:23,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:24,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:25,505 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:25,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:26,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:27,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:28,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:29,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:30,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:31,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:31,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:32,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:33,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:34,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:34,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:50:34,665 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:50:35,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:36,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:36,870 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:50:36,872 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:50:36,872 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:50:36,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:36,874 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:50:37,414 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:50:37,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:38,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:39,415 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:50:39,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:40,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:41,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:42,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:42,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:43,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:44,536 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:45,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:46,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:47,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:48,121 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:50:48,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:48,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:49,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:49,888 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:50:49,888 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:50:50,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:51,421 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:50:51,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:52,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:53,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:53,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:54,537 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:55,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:56,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:57,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:58,508 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:50:58,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:50:59,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:00,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:01,541 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:02,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:03,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:03,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:04,546 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:04,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:51:04,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:51:05,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:06,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:07,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:51:07,236 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:51:07,236 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:51:07,237 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:51:07,441 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:51:07,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:08,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:09,275 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:09,442 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:51:09,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:10,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:11,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:12,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:13,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:14,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:14,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:15,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:16,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:17,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:18,125 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:51:18,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:19,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:19,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:51:19,804 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:51:19,933 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:20,545 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:21,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:22,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:23,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:24,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:25,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:25,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:26,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:27,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:28,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:29,573 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:30,574 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:30,979 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:33,459 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:51:33,524 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:36,783 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:40,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:51:40,744 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:51:41,103 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:41,804 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:41,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:46,887 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:48,130 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:51:49,588 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:50,794 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:52,160 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:57,164 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:51:58,280 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:51:58,344 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:51:58,345 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:51:59,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:02,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:06,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:07,511 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:12,512 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:14,664 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:15,576 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:52:15,577 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:52:16,026 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:17,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:18,134 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:52:23,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:28,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:28,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:28,727 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:52:29,046 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:29,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:52:29,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:52:29,157 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:30,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:30,729 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:52:31,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:32,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:33,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:34,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:35,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:35,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:36,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:37,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:38,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:39,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:40,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:40,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:41,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:42,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:43,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:44,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:44,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:52:44,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:52:45,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:45,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:46,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:47,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:48,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:48,139 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:52:49,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:50,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:51,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:51,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:52,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:53,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:54,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:55,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:56,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:57,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:52:57,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:58,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:59,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:52:59,086 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:52:59,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:53:00,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:01,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:02,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:02,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:03,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:04,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:05,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:06,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:07,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:08,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:08,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:09,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:10,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:11,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:12,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:13,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:14,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:14,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:14,087 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:53:14,087 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:53:15,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:16,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:17,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:18,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:18,142 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:53:19,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:19,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:20,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:24,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:29,145 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:34,146 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:39,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:41,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:41,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:53:41,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:53:42,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:43,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:44,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:44,493 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:45,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:46,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:47,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:48,146 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:53:48,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:49,387 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:50,148 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:50,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:51,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:52,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:53,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:54,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:55,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:55,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:53:56,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:56,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:53:56,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:53:57,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:58,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:53:59,388 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:00,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:00,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:01,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:02,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:03,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:04,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:05,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:06,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:06,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:07,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:08,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:09,389 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:10,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:11,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:11,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:11,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:54:11,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:54:12,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:13,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:14,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:15,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:16,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:17,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:17,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:18,150 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:54:18,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:19,390 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:20,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:21,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:22,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:22,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:23,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:24,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:25,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:26,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:26,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:54:26,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:54:27,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:27,501 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:28,391 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:29,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:30,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:31,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:32,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:33,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:33,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:34,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:35,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:36,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:37,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:38,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:39,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:39,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:40,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:41,392 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:54:41,393 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:54:41,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:42,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:43,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:44,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:44,550 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:45,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:46,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:47,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:48,154 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:54:48,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:49,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:50,157 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:50,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:51,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:52,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:53,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:54,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:55,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:54:55,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:56,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:54:56,394 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:54:56,434 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:57,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:58,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:54:59,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:00,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:00,499 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:01,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:02,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:03,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:04,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:05,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:06,393 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:06,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:07,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:08,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:09,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:10,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:11,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:11,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:55:11,395 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:55:11,435 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:12,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:13,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:14,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:15,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:16,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:17,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:17,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:18,159 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:55:18,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:19,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:20,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:21,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:22,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:22,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:55:27,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:32,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:37,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:42,397 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:47,398 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:48,183 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:55:53,185 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:55:58,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:02,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:02,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:56:02,873 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:56:03,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:04,010 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:04,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:05,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:06,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:07,609 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:56:07,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:08,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:09,568 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:09,611 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:56:09,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:10,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:11,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:12,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:13,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:14,867 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:14,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:15,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:16,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:17,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:56:17,867 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:56:17,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:18,166 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:56:18,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:19,618 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:56:19,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:20,682 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:20,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:21,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:22,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:23,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:24,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:25,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:25,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:26,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:27,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:28,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:29,869 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:30,868 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:30,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:31,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:32,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:56:32,867 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:56:32,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:33,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:34,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:35,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:36,011 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:36,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:56:36,130 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:56:36,131 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:56:36,132 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:56:36,637 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:56:36,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:37,638 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:56:37,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:38,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:39,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:40,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:41,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:41,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:42,876 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:43,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:44,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:45,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:46,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:46,877 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:47,866 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:56:47,867 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:56:47,907 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:48,169 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:56:48,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:49,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:50,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:51,647 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:56:51,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:51,930 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:52,927 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:53,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:54,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:55,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:56,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:57,872 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:56:57,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:58,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:56:59,928 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:00,937 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:01,935 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:02,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:57:02,871 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:57:02,938 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:02,969 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:03,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:04,941 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:05,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:06,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:57:06,809 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:57:06,810 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:57:06,811 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:57:07,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:07,667 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:57:08,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:08,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:09,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:09,668 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:57:10,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:11,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:12,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:13,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:14,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:14,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:15,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:16,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:17,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:17,871 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:57:17,871 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:57:18,173 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:57:18,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:19,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:20,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:20,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:21,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:21,675 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:57:22,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:23,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:24,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:25,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:25,878 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:26,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:27,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:28,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:29,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:30,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:30,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:31,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:32,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:32,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:57:32,873 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:57:33,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:34,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:35,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:36,014 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:36,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:36,737 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:57:36,739 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:57:36,739 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:57:36,741 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:57:37,257 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:37,698 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:57:37,699 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:57:38,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:39,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:39,700 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:57:40,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:41,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:41,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:42,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:43,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:44,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:45,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:46,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:47,261 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:47,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:57:47,873 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:57:47,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:48,177 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:57:48,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:49,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:50,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:51,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:52,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:53,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:53,708 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:57:53,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:54,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:55,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:56,262 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:57,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:58,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:57:58,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:57:59,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:00,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:01,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:02,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:02,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:58:02,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:58:03,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:04,032 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:04,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:05,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:06,263 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:07,267 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:08,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:09,274 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:09,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:10,269 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:11,062 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:58:11,064 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:58:11,065 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:58:11,066 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:58:11,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:11,723 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:58:12,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:13,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:13,724 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:58:14,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:15,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:15,882 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:16,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:17,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:17,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:58:17,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:58:18,181 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:58:18,470 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:19,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:20,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:21,184 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:21,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:22,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:23,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:24,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:25,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:25,734 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:58:26,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:26,534 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:27,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:28,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:29,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:30,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:31,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:31,883 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:32,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:32,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:58:32,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:58:33,532 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:34,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:35,533 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:36,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:37,021 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:37,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:38,544 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:39,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:40,542 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:41,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:58:41,467 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:58:41,468 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:58:41,469 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:58:41,755 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:58:41,884 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:42,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:42,885 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:43,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:43,756 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:58:44,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:45,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:46,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:47,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:47,873 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:58:47,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:58:48,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:48,186 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:58:48,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:49,612 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:50,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:51,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:52,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:53,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:53,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:54,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:55,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:55,765 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:58:56,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:57,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:58,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:58:58,886 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:58:59,613 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:00,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:01,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:02,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:02,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:59:02,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:59:03,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:03,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:04,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:05,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:06,621 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:07,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:08,617 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:09,394 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:59:09,395 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:59:09,395 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:59:09,396 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:09,397 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:59:09,786 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:59:09,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:10,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:11,788 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:59:11,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:12,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:13,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:14,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:14,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:15,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:16,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:17,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:17,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:59:17,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:59:18,189 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:59:18,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:19,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:20,192 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:20,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:21,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:22,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:23,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:24,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:25,377 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:25,795 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:59:25,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:26,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:27,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:28,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:29,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:30,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:30,889 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:31,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:32,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:32,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:59:32,874 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:59:33,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:34,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:35,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:35,995 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:36,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:37,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:38,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:39,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:40,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:41,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:41,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:42,080 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 14:59:42,082 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 14:59:42,082 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 14:59:42,083 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 14:59:42,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:42,816 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 14:59:43,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:43,817 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:59:44,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:45,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:46,811 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:47,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:47,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 14:59:47,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 14:59:48,025 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:48,193 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 14:59:48,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:49,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:50,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:51,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:52,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:53,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:53,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:54,812 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:55,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:56,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:57,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:57,826 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 14:59:58,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 14:59:58,893 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 14:59:59,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:00,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:01,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:02,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:02,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:00:02,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:00:03,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:03,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:04,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:05,813 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:06,814 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:07,817 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:08,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:09,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:09,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:10,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:11,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:12,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:13,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:00:13,121 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:00:13,121 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:00:13,123 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:00:13,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:13,841 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:00:14,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:15,529 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:15,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:15,843 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:00:16,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:17,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:17,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:00:17,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:00:18,198 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:00:18,822 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:19,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:20,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:21,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:21,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:22,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:23,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:24,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:25,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:26,417 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:26,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:27,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:27,851 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:00:28,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:29,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:30,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:31,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:31,899 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:32,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:32,874 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:00:32,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:00:33,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:34,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:35,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:36,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:36,981 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:37,824 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:38,825 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:39,828 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:40,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:41,832 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:42,835 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:42,901 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:43,843 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:44,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:45,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:00:45,143 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:00:45,144 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:00:45,145 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:00:45,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:45,874 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:00:46,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:47,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:47,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:00:47,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:00:47,881 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:00:48,015 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:48,202 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:00:48,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:49,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:50,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:51,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:52,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:53,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:53,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:54,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:55,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:56,836 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:57,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:58,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:58,903 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:00:59,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:00:59,889 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:01:00,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:01,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:02,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:02,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:01:02,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:01:03,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:03,975 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:04,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:05,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:06,837 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:07,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:08,838 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:09,849 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:09,909 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:10,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:11,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:12,847 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:13,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:14,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:14,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:15,408 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:01:15,410 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:01:15,410 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:01:15,411 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:01:15,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:15,906 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:01:16,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:17,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:17,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:01:17,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:01:17,908 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:01:18,207 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:01:18,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:19,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:20,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:20,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:21,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:22,851 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:23,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:24,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:25,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:25,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:26,852 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:28,082 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:28,942 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:29,915 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:01:29,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:30,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:31,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:31,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:32,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:01:32,876 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:01:32,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:33,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:34,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:35,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:36,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:37,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:37,943 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:38,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:39,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:40,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:41,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:42,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:42,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:43,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:44,948 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:45,806 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:01:45,807 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:01:45,807 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:01:45,809 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:01:45,935 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:01:46,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:47,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:47,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:47,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:01:47,875 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:01:47,936 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:01:48,210 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:01:48,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:49,218 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:50,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:51,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:52,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:53,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:53,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:54,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:55,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:56,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:57,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:58,219 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:01:59,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:01:59,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:00,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:01,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:01,945 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:02:02,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:02,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:02:02,876 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:02:03,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:04,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:05,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:05,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:06,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:07,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:08,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:09,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:10,044 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:10,297 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:11,306 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:12,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:13,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:14,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:15,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:15,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:16,302 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:17,242 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:02:17,243 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:02:17,244 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:02:17,245 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:02:17,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:17,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:02:17,876 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:02:17,963 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:02:17,963 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:02:18,214 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:02:18,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:19,649 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:19,965 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:02:20,216 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:20,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:21,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:22,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:23,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:24,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:25,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:26,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:26,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:27,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:28,650 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:30,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:31,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:31,101 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:31,972 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:02:32,098 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:32,875 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:02:32,876 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:02:33,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:34,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:35,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:36,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:37,047 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:37,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:38,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:39,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:40,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:41,099 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:42,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:42,108 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:43,104 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:44,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:45,106 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:46,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:47,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:47,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:47,620 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:02:47,621 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:02:47,621 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:02:47,623 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:02:47,994 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:02:48,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:02:48,038 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:02:48,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:48,218 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:02:49,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:49,995 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:02:50,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:51,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:52,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:52,220 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:53,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:54,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:55,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:56,111 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:57,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:58,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:02:58,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:02:59,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:00,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:01,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:02,002 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:03:02,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:03,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:03:03,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:03:03,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:03,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:04,112 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:05,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:06,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:07,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:08,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:09,052 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:09,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:10,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:11,113 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:12,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:13,119 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:14,061 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:14,126 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:15,121 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:16,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:17,122 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:18,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:03:18,038 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:03:18,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:03:18,123 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:03:18,124 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:03:18,126 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:03:18,222 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:03:18,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:19,024 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:03:19,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:19,468 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:20,025 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:03:20,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:21,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:22,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:23,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:24,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:25,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:25,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:26,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:27,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:28,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:29,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:30,466 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:31,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:31,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:32,033 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:03:32,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:33,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:03:33,037 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:03:33,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:34,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:35,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:36,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:36,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:37,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:38,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:39,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:40,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:41,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:42,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:42,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:43,471 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:44,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:45,475 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:46,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:47,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:47,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:48,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:03:48,041 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:03:48,228 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:03:48,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:48,859 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:03:48,860 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:03:48,860 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:03:48,862 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:03:49,053 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:03:49,472 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:50,054 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:03:50,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:51,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:52,268 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:52,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:53,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:54,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:55,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:56,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:57,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:58,055 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:03:58,477 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:03:59,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:00,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:01,478 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:02,675 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:03,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:04:03,041 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:04:03,166 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:03,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:04,065 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:04:04,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:05,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:06,584 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:07,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:08,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:09,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:09,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:10,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:11,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:12,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:13,585 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:14,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:15,056 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:15,592 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:16,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:17,596 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:18,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:04:18,055 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:04:18,233 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:04:18,593 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:19,595 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:20,167 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:04:20,168 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:04:20,169 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:04:20,169 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:20,171 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:04:20,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:21,088 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:04:21,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:22,089 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:04:22,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:23,687 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:24,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:25,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:26,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:26,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:27,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:28,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:29,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:30,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:31,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:32,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:32,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:33,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:04:33,045 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:04:33,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:34,688 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:35,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:36,097 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:04:36,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:37,538 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:37,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:38,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:39,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:40,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:41,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:42,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:43,058 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:43,689 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:44,690 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:45,695 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:46,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:47,700 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:48,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:04:48,049 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:04:48,193 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:48,246 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:04:48,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:49,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:50,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:50,906 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:04:50,908 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:04:50,908 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:04:50,909 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:04:51,122 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:04:51,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:52,123 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:04:52,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:53,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:54,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:04:54,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:55,697 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:56,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:57,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:58,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:04:59,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:00,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:00,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:01,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:02,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:03,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:05:03,049 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:05:03,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:04,130 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:05:04,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:05,214 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:05,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:06,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:07,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:08,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:09,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:10,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:11,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:11,699 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:12,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:13,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:14,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:15,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:16,071 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:16,706 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:17,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:17,983 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:05:17,984 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:05:17,984 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:05:17,988 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:05:18,146 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:05:18,242 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:05:18,395 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:05:18,396 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:05:18,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:19,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:20,147 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:05:20,702 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:21,496 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:21,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:22,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:23,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:24,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:25,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:26,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:27,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:27,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:28,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:29,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:30,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:31,703 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:32,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:32,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:33,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:05:33,124 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:05:33,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:34,156 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:05:34,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:35,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:36,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:37,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:37,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:38,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:39,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:40,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:41,704 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:42,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:43,124 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:43,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:44,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:45,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:46,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:47,714 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:48,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:05:48,128 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:05:48,272 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:05:48,273 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:48,713 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:49,715 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:50,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:51,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:52,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:05:52,182 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:05:52,182 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:05:52,183 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:05:52,711 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:53,181 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:05:53,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:54,127 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:54,182 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:05:54,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:55,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:56,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:57,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:58,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:05:59,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:05:59,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:00,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:01,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:02,712 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:03,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:06:03,125 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:06:03,890 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:04,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:04,891 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:05,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:06,190 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:06:06,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:07,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:08,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:09,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:10,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:10,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:11,798 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:12,799 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:13,815 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:14,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:15,134 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:15,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:16,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:17,803 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:18,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:06:18,131 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:06:18,249 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:06:18,802 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:19,626 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:06:19,627 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:06:19,628 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:06:19,633 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:06:20,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:20,207 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:06:21,036 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:21,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:22,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:22,209 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:06:23,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:24,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:25,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:26,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:26,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:27,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:28,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:29,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:30,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:31,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:32,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:32,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:33,037 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:33,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:06:33,125 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:06:34,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:35,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:36,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:36,218 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:06:37,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:37,432 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:38,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:39,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:40,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:41,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:42,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:43,038 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:43,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:44,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:45,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:46,039 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:47,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:48,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:48,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:06:48,130 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:06:48,286 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:06:48,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:49,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:50,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:51,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:52,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:53,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:53,067 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:06:53,068 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:06:53,069 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:06:53,074 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:06:53,236 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:06:53,467 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:06:54,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:54,237 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:06:55,040 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:56,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:57,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:58,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:59,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:06:59,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:00,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:01,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:02,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:03,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:03,125 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:07:03,126 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:07:04,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:04,290 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:05,041 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:06,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:07,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:08,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:08,246 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:07:09,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:10,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:10,212 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:11,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:12,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:13,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:14,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:15,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:16,042 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:16,213 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:17,043 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:18,054 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:18,129 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:07:18,135 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:07:18,281 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:07:19,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:20,051 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:21,048 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:21,285 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:22,050 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:23,049 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:23,767 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:07:23,768 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:07:23,768 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:07:23,773 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:07:24,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:24,267 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:07:25,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:26,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:26,269 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:07:27,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:27,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:28,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:29,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:30,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:31,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:32,181 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:32,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:33,128 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:07:33,129 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:07:33,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:34,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:35,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:36,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:37,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:37,998 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:38,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:38,277 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:07:39,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:40,182 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:41,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:42,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:43,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:43,217 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:44,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:45,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:46,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:47,183 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:48,136 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:07:48,141 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:07:48,189 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:48,308 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:07:48,315 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:49,188 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:50,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:51,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:52,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:53,195 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:54,178 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:07:54,179 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:07:54,180 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:07:54,180 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:54,183 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:07:54,297 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:07:54,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:55,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:56,299 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:07:56,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:57,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:58,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:07:59,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:07:59,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:00,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:01,589 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:02,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:03,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:08:03,133 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:08:03,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:04,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:04,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:05,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:06,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:07,590 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:08,696 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:09,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:09,698 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:10,307 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:08:10,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:11,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:12,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:13,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:14,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:15,223 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:15,609 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:16,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:17,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:18,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:08:18,133 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:08:18,298 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:08:18,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:19,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:20,301 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:20,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:21,610 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:22,618 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:23,615 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:24,614 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:25,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:26,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:26,616 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:27,623 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:28,607 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:08:28,608 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:08:28,609 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:08:28,613 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:08:29,324 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:08:29,805 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:29,846 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:30,325 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:08:30,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:31,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:31,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:32,807 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:33,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:08:33,134 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:08:33,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:34,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:35,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:36,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:37,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:37,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:38,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:39,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:40,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:41,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:42,507 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:42,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:43,808 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:44,333 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:08:44,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:45,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:46,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:47,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:48,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:08:48,134 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:08:48,245 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:48,271 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:08:48,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:49,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:50,809 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:51,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:52,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:53,810 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:54,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:54,823 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:55,821 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:56,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:57,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:58,818 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:08:59,227 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:08:59,819 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:00,539 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:09:00,541 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:09:00,541 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:09:00,547 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:09:00,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:01,357 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:09:01,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:02,358 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:09:02,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:03,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:09:03,134 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:09:03,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:04,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:04,944 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:05,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:06,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:07,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:08,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:09,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:10,224 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:10,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:11,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:12,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:13,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:14,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:15,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:15,945 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:16,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:17,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:18,133 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:09:18,134 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:09:18,275 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:09:18,367 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:09:18,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:19,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:20,277 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:20,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:21,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:22,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:23,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:24,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:25,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:26,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:26,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:27,946 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:28,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:29,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:30,947 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:31,233 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:31,949 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:32,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:33,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:09:33,143 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:09:33,958 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:34,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:35,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:36,270 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:36,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:36,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:09:36,964 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:09:36,964 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:09:36,968 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:09:37,401 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:09:37,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:38,402 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:09:38,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:39,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:40,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:41,950 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:42,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:42,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:43,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:44,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:45,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:46,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:47,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:48,141 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:09:48,141 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:09:48,254 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:48,279 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:09:48,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:49,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:50,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:51,951 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:52,409 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:09:52,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:53,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:54,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:09:54,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:55,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:56,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:57,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:58,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:09:59,952 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:00,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:00,953 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:01,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:02,955 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:03,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:10:03,147 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:10:03,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:04,959 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:05,276 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:05,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:06,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:07,154 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:10:07,156 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:10:07,156 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:10:07,159 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:10:07,427 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:10:07,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:08,428 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:10:08,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:09,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:10,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:11,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:11,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:12,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:13,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:14,961 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:15,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:16,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:17,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:17,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:18,144 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:10:18,144 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:10:18,283 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:10:18,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:19,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:20,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:21,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:22,304 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:22,437 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:10:22,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:23,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:24,962 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:25,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:26,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:27,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:28,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:28,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:29,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:30,963 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:31,974 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:32,969 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:33,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:10:33,147 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:10:33,969 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:34,229 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:34,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:35,976 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:36,973 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:37,648 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:10:37,649 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:10:37,649 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:10:37,651 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:10:38,066 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:38,466 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:10:38,466 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:10:39,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:39,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:40,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:40,467 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:10:41,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:42,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:43,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:44,063 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:44,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:45,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:46,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:47,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:48,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:48,147 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:10:48,147 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:10:48,287 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:10:49,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:49,289 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:50,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:51,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:52,064 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:52,474 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:10:53,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:54,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:55,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:55,230 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:10:56,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:57,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:58,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:10:59,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:00,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:01,065 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:01,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:02,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:03,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:03,156 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:11:03,156 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:11:04,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:05,074 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:06,077 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:06,240 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:07,053 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:11:07,054 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:11:07,054 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:11:07,056 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:11:07,462 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:07,496 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:11:08,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:08,496 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:11:09,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:10,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:11,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:12,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:12,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:13,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:14,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:15,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:16,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:17,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:18,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:11:18,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:11:18,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:18,293 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:11:18,293 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:19,196 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:20,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:21,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:22,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:22,505 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:11:23,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:24,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:24,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:25,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:26,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:27,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:28,197 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:29,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:29,234 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:30,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:31,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:32,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:33,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:11:33,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:11:33,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:34,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:34,246 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:35,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:36,198 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:37,203 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:38,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:39,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:40,199 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:40,239 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:40,364 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:11:40,367 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:11:40,367 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:11:40,372 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:11:40,535 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:11:41,205 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:42,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:42,536 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:11:43,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:44,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:45,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:46,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:46,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:47,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:48,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:11:48,151 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:11:48,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:48,294 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:11:49,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:50,200 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:51,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:51,296 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:52,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:53,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:54,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:54,543 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:11:55,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:56,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:57,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:57,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:11:58,201 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:11:59,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:00,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:01,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:02,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:02,237 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:03,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:12:03,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:12:03,202 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:04,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:05,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:06,209 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:07,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:07,279 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:08,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:09,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:09,337 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:12:09,338 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:12:09,338 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:12:09,340 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:12:09,560 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:12:10,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:10,561 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:12:11,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:12,206 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:12,749 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:13,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:14,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:15,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:16,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:17,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:18,151 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:12:18,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:12:18,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:18,310 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:12:18,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:19,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:20,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:21,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:22,207 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:23,208 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:24,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:24,318 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:25,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:26,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:26,570 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:12:27,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:28,225 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:29,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:29,243 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:30,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:31,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:32,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:33,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:12:33,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:12:33,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:34,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:34,295 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:35,226 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:36,235 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:37,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:38,228 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:39,231 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:40,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:40,251 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:41,232 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:42,028 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:12:42,029 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:12:42,030 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:12:42,031 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:12:42,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:42,591 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:12:42,591 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:12:43,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:44,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:44,592 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:12:45,436 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:46,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:46,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:47,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:48,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:12:48,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:12:48,308 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:12:48,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:49,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:50,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:51,310 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:51,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:52,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:53,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:54,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:54,598 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:12:55,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:56,437 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:57,045 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:12:57,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:58,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:12:59,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:00,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:01,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:02,247 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:02,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:03,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:13:03,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:13:03,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:04,438 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:05,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:06,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:07,281 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:07,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:08,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:09,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:10,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:11,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:11,491 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:13:11,493 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:13:11,493 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:13:11,495 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:13:11,616 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:13:12,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:12,617 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:13:12,904 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:13,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:14,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:15,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:16,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:17,439 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:18,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:13:18,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:13:18,286 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:18,306 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:13:18,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:19,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:20,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:21,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:22,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:23,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:24,248 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:24,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:25,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:26,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:27,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:28,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:28,628 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:13:29,249 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:29,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:30,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:31,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:32,440 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:33,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:13:33,152 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:13:33,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:34,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:34,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:35,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:36,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:37,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:38,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:39,442 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:40,255 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:40,441 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:41,447 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:42,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:43,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:44,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:45,259 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:45,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:45,483 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:13:45,484 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:13:45,485 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:13:45,487 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:13:45,651 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:13:46,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:46,653 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:13:47,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:48,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:13:48,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:13:48,310 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:13:48,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:49,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:50,312 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:50,448 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:51,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:52,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:53,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:54,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:55,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:56,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:13:56,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:57,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:58,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:13:58,661 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:13:59,449 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:00,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:01,253 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:01,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:02,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:03,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:14:03,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:14:03,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:04,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:05,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:06,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:06,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:07,450 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:08,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:09,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:10,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:11,454 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:12,271 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:12,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:13,455 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:14,334 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:14:14,336 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:14:14,336 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:14:14,338 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:14:14,681 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:14:14,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:15,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:16,682 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:14:16,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:17,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:17,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:18,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:14:18,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:14:18,314 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:14:18,740 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:19,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:20,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:21,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:22,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:23,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:23,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:24,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:25,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:26,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:27,741 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:28,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:29,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:29,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:30,690 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:14:30,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:31,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:32,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:33,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:14:33,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:14:33,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:34,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:34,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:35,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:36,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:37,742 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:38,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:39,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:40,258 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:40,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:41,743 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:42,747 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:43,744 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:44,752 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:45,266 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:45,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:46,753 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:47,750 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:48,152 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: stop_status 2025-01-02 15:14:48,153 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: stop_status 2025-01-02 15:14:48,318 DEBUG SenderThread:2914664 [sender.py:send():379] send: stats 2025-01-02 15:14:48,705 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: partial_history 2025-01-02 15:14:48,707 DEBUG SenderThread:2914664 [sender.py:send():379] send: history 2025-01-02 15:14:48,707 DEBUG SenderThread:2914664 [sender.py:send_request():406] send_request: summary_record 2025-01-02 15:14:48,709 INFO SenderThread:2914664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-02 15:14:49,114 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:49,709 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/wandb-summary.json 2025-01-02 15:14:49,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:50,710 INFO Thread-12 :2914664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250102_074844-1ecgrehs/files/output.log 2025-01-02 15:14:50,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:51,116 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report 2025-01-02 15:14:51,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:52,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:53,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:54,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:55,850 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-02 15:14:56,260 DEBUG HandlerThread:2914664 [handler.py:handle_request():158] handle_request: status_report