Upload folder using huggingface_hub

README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:080871d19dd4c4011cfd878b0a1e5fdab7625d2e082b58e4c2ad35dacdc85506
+size 25172088

checkpoint-3125/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

checkpoint-3125/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-3125/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:080871d19dd4c4011cfd878b0a1e5fdab7625d2e082b58e4c2ad35dacdc85506
+size 25172088

checkpoint-3125/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dec7a5cdeb696d3de2c0b8be7ca6c571d8a9a174dfeed727a849c7a542aa4ec3
+size 50372538

checkpoint-3125/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5994634e90ce1b45d012a45568063be05fea876e791cd66b48a4efc924164b2
+size 14244

checkpoint-3125/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e42e57efa8b95889e87232fe6ea0805483ed6c8a2d35bb60230715b624ff5bda
+size 1064

checkpoint-3125/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-3125/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-3125/trainer_state.json

@@ -0,0 +1,2217 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 3125,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 1.0,
+      "learning_rate": 6.369426751592357e-06,
+      "loss": 22.4391,
+      "step": 10
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 1.0,
+      "learning_rate": 1.2738853503184714e-05,
+      "loss": 18.54,
+      "step": 20
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 1.0,
+      "learning_rate": 1.910828025477707e-05,
+      "loss": 18.87,
+      "step": 30
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 2.5477707006369428e-05,
+      "loss": 17.984,
+      "step": 40
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 3.184713375796178e-05,
+      "loss": 15.0193,
+      "step": 50
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 3.821656050955414e-05,
+      "loss": 14.2766,
+      "step": 60
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 1.0000001192092896,
+      "learning_rate": 4.45859872611465e-05,
+      "loss": 11.9355,
+      "step": 70
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 5.0955414012738855e-05,
+      "loss": 9.6631,
+      "step": 80
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 5.732484076433121e-05,
+      "loss": 8.2441,
+      "step": 90
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 6.369426751592356e-05,
+      "loss": 7.0992,
+      "step": 100
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 7.006369426751592e-05,
+      "loss": 5.2436,
+      "step": 110
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 7.643312101910829e-05,
+      "loss": 5.0585,
+      "step": 120
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 8.280254777070065e-05,
+      "loss": 4.3211,
+      "step": 130
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 8.9171974522293e-05,
+      "loss": 3.745,
+      "step": 140
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.554140127388536e-05,
+      "loss": 3.5158,
+      "step": 150
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.989892183288411e-05,
+      "loss": 3.2362,
+      "step": 160
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.956199460916442e-05,
+      "loss": 2.8799,
+      "step": 170
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 9.922506738544474e-05,
+      "loss": 2.7697,
+      "step": 180
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 9.888814016172507e-05,
+      "loss": 2.7384,
+      "step": 190
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.85512129380054e-05,
+      "loss": 2.4178,
+      "step": 200
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.821428571428572e-05,
+      "loss": 2.3849,
+      "step": 210
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.787735849056603e-05,
+      "loss": 2.1359,
+      "step": 220
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.754043126684636e-05,
+      "loss": 2.2128,
+      "step": 230
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.720350404312669e-05,
+      "loss": 2.12,
+      "step": 240
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.686657681940702e-05,
+      "loss": 1.9436,
+      "step": 250
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.652964959568734e-05,
+      "loss": 1.9452,
+      "step": 260
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.619272237196765e-05,
+      "loss": 1.9896,
+      "step": 270
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.585579514824798e-05,
+      "loss": 1.8534,
+      "step": 280
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 9.551886792452831e-05,
+      "loss": 1.5842,
+      "step": 290
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.518194070080863e-05,
+      "loss": 1.8386,
+      "step": 300
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.484501347708896e-05,
+      "loss": 1.7605,
+      "step": 310
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.450808625336927e-05,
+      "loss": 1.6376,
+      "step": 320
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.41711590296496e-05,
+      "loss": 1.4613,
+      "step": 330
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 9.383423180592993e-05,
+      "loss": 1.5167,
+      "step": 340
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.349730458221025e-05,
+      "loss": 1.5203,
+      "step": 350
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.316037735849057e-05,
+      "loss": 1.3776,
+      "step": 360
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.282345013477089e-05,
+      "loss": 1.3961,
+      "step": 370
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.248652291105122e-05,
+      "loss": 1.3971,
+      "step": 380
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.214959568733154e-05,
+      "loss": 1.2078,
+      "step": 390
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.181266846361186e-05,
+      "loss": 1.1744,
+      "step": 400
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 9.14757412398922e-05,
+      "loss": 1.1205,
+      "step": 410
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 9.113881401617251e-05,
+      "loss": 1.0612,
+      "step": 420
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.080188679245284e-05,
+      "loss": 1.0844,
+      "step": 430
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 9.046495956873315e-05,
+      "loss": 1.1055,
+      "step": 440
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 9.012803234501348e-05,
+      "loss": 1.0199,
+      "step": 450
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.979110512129381e-05,
+      "loss": 0.987,
+      "step": 460
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.945417789757413e-05,
+      "loss": 0.9312,
+      "step": 470
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 8.911725067385444e-05,
+      "loss": 0.9257,
+      "step": 480
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.878032345013477e-05,
+      "loss": 0.828,
+      "step": 490
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.84433962264151e-05,
+      "loss": 0.9471,
+      "step": 500
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.810646900269543e-05,
+      "loss": 0.8311,
+      "step": 510
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.776954177897575e-05,
+      "loss": 0.8505,
+      "step": 520
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 8.743261455525606e-05,
+      "loss": 0.7594,
+      "step": 530
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 8.709568733153639e-05,
+      "loss": 0.817,
+      "step": 540
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.675876010781672e-05,
+      "loss": 0.7966,
+      "step": 550
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 8.642183288409704e-05,
+      "loss": 0.7625,
+      "step": 560
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 8.608490566037735e-05,
+      "loss": 0.724,
+      "step": 570
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.574797843665768e-05,
+      "loss": 0.7257,
+      "step": 580
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.541105121293801e-05,
+      "loss": 0.7533,
+      "step": 590
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.507412398921834e-05,
+      "loss": 0.707,
+      "step": 600
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.473719676549866e-05,
+      "loss": 0.66,
+      "step": 610
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 8.440026954177897e-05,
+      "loss": 0.6193,
+      "step": 620
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 8.40633423180593e-05,
+      "loss": 0.6132,
+      "step": 630
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 8.372641509433963e-05,
+      "loss": 0.6331,
+      "step": 640
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.338948787061996e-05,
+      "loss": 0.5495,
+      "step": 650
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.305256064690027e-05,
+      "loss": 0.5478,
+      "step": 660
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 8.271563342318059e-05,
+      "loss": 0.5972,
+      "step": 670
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 8.237870619946092e-05,
+      "loss": 0.5714,
+      "step": 680
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.204177897574125e-05,
+      "loss": 0.5494,
+      "step": 690
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.170485175202158e-05,
+      "loss": 0.5474,
+      "step": 700
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.136792452830189e-05,
+      "loss": 0.543,
+      "step": 710
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 8.103099730458221e-05,
+      "loss": 0.5209,
+      "step": 720
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.069407008086254e-05,
+      "loss": 0.509,
+      "step": 730
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 8.035714285714287e-05,
+      "loss": 0.555,
+      "step": 740
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 8.002021563342318e-05,
+      "loss": 0.4649,
+      "step": 750
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.968328840970351e-05,
+      "loss": 0.4629,
+      "step": 760
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.934636118598383e-05,
+      "loss": 0.4924,
+      "step": 770
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 7.900943396226416e-05,
+      "loss": 0.4953,
+      "step": 780
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.867250673854449e-05,
+      "loss": 0.507,
+      "step": 790
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.83355795148248e-05,
+      "loss": 0.5165,
+      "step": 800
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.799865229110512e-05,
+      "loss": 0.4489,
+      "step": 810
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 7.766172506738545e-05,
+      "loss": 0.4279,
+      "step": 820
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 7.732479784366577e-05,
+      "loss": 0.4258,
+      "step": 830
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.69878706199461e-05,
+      "loss": 0.4614,
+      "step": 840
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 7.665094339622642e-05,
+      "loss": 0.4691,
+      "step": 850
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.631401617250674e-05,
+      "loss": 0.4586,
+      "step": 860
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 7.597708894878706e-05,
+      "loss": 0.4347,
+      "step": 870
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.56401617250674e-05,
+      "loss": 0.4125,
+      "step": 880
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 7.530323450134771e-05,
+      "loss": 0.3699,
+      "step": 890
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 7.496630727762804e-05,
+      "loss": 0.441,
+      "step": 900
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 7.462938005390835e-05,
+      "loss": 0.4196,
+      "step": 910
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.429245283018868e-05,
+      "loss": 0.36,
+      "step": 920
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 7.395552560646901e-05,
+      "loss": 0.3673,
+      "step": 930
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 7.361859838274933e-05,
+      "loss": 0.3624,
+      "step": 940
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.328167115902966e-05,
+      "loss": 0.4013,
+      "step": 950
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 7.294474393530997e-05,
+      "loss": 0.3394,
+      "step": 960
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 7.26078167115903e-05,
+      "loss": 0.3839,
+      "step": 970
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 7.227088948787062e-05,
+      "loss": 0.3663,
+      "step": 980
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.193396226415095e-05,
+      "loss": 0.3547,
+      "step": 990
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 7.159703504043128e-05,
+      "loss": 0.377,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 7.126010781671159e-05,
+      "loss": 0.3347,
+      "step": 1010
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 0.9999998807907104,
+      "learning_rate": 7.092318059299192e-05,
+      "loss": 0.3758,
+      "step": 1020
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 7.058625336927224e-05,
+      "loss": 0.3692,
+      "step": 1030
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 7.024932614555257e-05,
+      "loss": 0.3215,
+      "step": 1040
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 6.99123989218329e-05,
+      "loss": 0.3417,
+      "step": 1050
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 6.957547169811321e-05,
+      "loss": 0.3101,
+      "step": 1060
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 6.923854447439353e-05,
+      "loss": 0.3272,
+      "step": 1070
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 6.890161725067386e-05,
+      "loss": 0.368,
+      "step": 1080
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 6.856469002695418e-05,
+      "loss": 0.3303,
+      "step": 1090
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 6.822776280323451e-05,
+      "loss": 0.2986,
+      "step": 1100
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 0.9999994039535522,
+      "learning_rate": 6.789083557951483e-05,
+      "loss": 0.2911,
+      "step": 1110
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 6.755390835579514e-05,
+      "loss": 0.3068,
+      "step": 1120
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 6.721698113207547e-05,
+      "loss": 0.268,
+      "step": 1130
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 6.68800539083558e-05,
+      "loss": 0.2994,
+      "step": 1140
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 6.654312668463612e-05,
+      "loss": 0.2921,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 0.9999998211860657,
+      "learning_rate": 6.620619946091643e-05,
+      "loss": 0.2631,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 6.586927223719676e-05,
+      "loss": 0.2519,
+      "step": 1170
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 6.553234501347709e-05,
+      "loss": 0.2571,
+      "step": 1180
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 0.9999999403953552,
+      "learning_rate": 6.519541778975742e-05,
+      "loss": 0.2621,
+      "step": 1190
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 6.485849056603774e-05,
+      "loss": 0.2332,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 6.452156334231805e-05,
+      "loss": 0.2531,
+      "step": 1210
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 6.418463611859838e-05,
+      "loss": 0.2334,
+      "step": 1220
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 6.384770889487871e-05,
+      "loss": 0.2208,
+      "step": 1230
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 6.351078167115904e-05,
+      "loss": 0.234,
+      "step": 1240
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 6.317385444743936e-05,
+      "loss": 0.2321,
+      "step": 1250
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 6.283692722371967e-05,
+      "loss": 0.2057,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 0.9999994039535522,
+      "learning_rate": 6.25e-05,
+      "loss": 0.2129,
+      "step": 1270
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 6.216307277628033e-05,
+      "loss": 0.2195,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 6.182614555256066e-05,
+      "loss": 0.2309,
+      "step": 1290
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 6.148921832884098e-05,
+      "loss": 0.1961,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 6.115229110512129e-05,
+      "loss": 0.2059,
+      "step": 1310
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 6.081536388140162e-05,
+      "loss": 0.1896,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 0.9999997615814209,
+      "learning_rate": 6.047843665768195e-05,
+      "loss": 0.1952,
+      "step": 1330
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 0.9999994039535522,
+      "learning_rate": 6.0141509433962265e-05,
+      "loss": 0.1865,
+      "step": 1340
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 5.980458221024259e-05,
+      "loss": 0.1781,
+      "step": 1350
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.9467654986522916e-05,
+      "loss": 0.1965,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 5.913072776280324e-05,
+      "loss": 0.1563,
+      "step": 1370
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 5.879380053908357e-05,
+      "loss": 0.1561,
+      "step": 1380
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 5.8456873315363884e-05,
+      "loss": 0.1601,
+      "step": 1390
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.8119946091644206e-05,
+      "loss": 0.1607,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 5.7783018867924535e-05,
+      "loss": 0.1637,
+      "step": 1410
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.744609164420486e-05,
+      "loss": 0.1591,
+      "step": 1420
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 5.710916442048517e-05,
+      "loss": 0.1782,
+      "step": 1430
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 5.6772237196765496e-05,
+      "loss": 0.1434,
+      "step": 1440
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 5.6435309973045825e-05,
+      "loss": 0.1356,
+      "step": 1450
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 5.609838274932615e-05,
+      "loss": 0.1321,
+      "step": 1460
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 0.9999994039535522,
+      "learning_rate": 5.5761455525606476e-05,
+      "loss": 0.1417,
+      "step": 1470
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.542452830188679e-05,
+      "loss": 0.1444,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.5087601078167114e-05,
+      "loss": 0.1284,
+      "step": 1490
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 5.4750673854447444e-05,
+      "loss": 0.1241,
+      "step": 1500
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 5.4413746630727766e-05,
+      "loss": 0.1312,
+      "step": 1510
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.407681940700808e-05,
+      "loss": 0.1299,
+      "step": 1520
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 5.373989218328841e-05,
+      "loss": 0.1332,
+      "step": 1530
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 5.340296495956873e-05,
+      "loss": 0.1232,
+      "step": 1540
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 5.306603773584906e-05,
+      "loss": 0.1147,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 5.2729110512129385e-05,
+      "loss": 0.1284,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 5.23921832884097e-05,
+      "loss": 0.1233,
+      "step": 1570
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 5.205525606469003e-05,
+      "loss": 0.1134,
+      "step": 1580
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 5.171832884097035e-05,
+      "loss": 0.1194,
+      "step": 1590
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.138140161725068e-05,
+      "loss": 0.1188,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 5.1044474393531e-05,
+      "loss": 0.1333,
+      "step": 1610
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 5.070754716981132e-05,
+      "loss": 0.143,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 5.037061994609165e-05,
+      "loss": 0.1038,
+      "step": 1630
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 5.003369272237197e-05,
+      "loss": 0.1689,
+      "step": 1640
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 4.969676549865229e-05,
+      "loss": 0.1065,
+      "step": 1650
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 4.9359838274932616e-05,
+      "loss": 0.1033,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 4.902291105121294e-05,
+      "loss": 0.1248,
+      "step": 1670
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 4.868598382749327e-05,
+      "loss": 0.1068,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 4.834905660377358e-05,
+      "loss": 0.0982,
+      "step": 1690
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 4.801212938005391e-05,
+      "loss": 0.0901,
+      "step": 1700
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 4.7675202156334234e-05,
+      "loss": 0.0866,
+      "step": 1710
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 4.733827493261456e-05,
+      "loss": 0.0891,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 4.7001347708894886e-05,
+      "loss": 0.0946,
+      "step": 1730
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 4.66644204851752e-05,
+      "loss": 0.1158,
+      "step": 1740
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 4.632749326145553e-05,
+      "loss": 0.1039,
+      "step": 1750
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 0.810341477394104,
+      "learning_rate": 4.5990566037735846e-05,
+      "loss": 0.0891,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 4.5653638814016176e-05,
+      "loss": 0.0821,
+      "step": 1770
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 0.9601090550422668,
+      "learning_rate": 4.53167115902965e-05,
+      "loss": 0.0895,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 4.497978436657682e-05,
+      "loss": 0.0987,
+      "step": 1790
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 4.464285714285715e-05,
+      "loss": 0.0932,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 4.4305929919137465e-05,
+      "loss": 0.084,
+      "step": 1810
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 4.3969002695417794e-05,
+      "loss": 0.0772,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 0.9272804856300354,
+      "learning_rate": 4.363207547169812e-05,
+      "loss": 0.08,
+      "step": 1830
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 4.329514824797844e-05,
+      "loss": 0.0849,
+      "step": 1840
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 0.9999990463256836,
+      "learning_rate": 4.295822102425876e-05,
+      "loss": 0.0845,
+      "step": 1850
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 4.2621293800539084e-05,
+      "loss": 0.0706,
+      "step": 1860
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 4.2284366576819406e-05,
+      "loss": 0.0763,
+      "step": 1870
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 4.1947439353099736e-05,
+      "loss": 0.0847,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 0.9999995827674866,
+      "learning_rate": 4.161051212938006e-05,
+      "loss": 0.0753,
+      "step": 1890
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 4.127358490566038e-05,
+      "loss": 0.0799,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 4.09366576819407e-05,
+      "loss": 0.0799,
+      "step": 1910
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 0.9999997019767761,
+      "learning_rate": 4.0599730458221025e-05,
+      "loss": 0.0799,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 0.9999989867210388,
+      "learning_rate": 4.026280323450135e-05,
+      "loss": 0.076,
+      "step": 1930
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 3.992587601078167e-05,
+      "loss": 0.0669,
+      "step": 1940
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 0.7363486289978027,
+      "learning_rate": 3.9588948787062e-05,
+      "loss": 0.0699,
+      "step": 1950
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 3.9252021563342315e-05,
+      "loss": 0.0716,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 0.9999989867210388,
+      "learning_rate": 3.8915094339622644e-05,
+      "loss": 0.0724,
+      "step": 1970
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 0.9999994039535522,
+      "learning_rate": 3.8578167115902966e-05,
+      "loss": 0.0717,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 3.824123989218329e-05,
+      "loss": 0.0669,
+      "step": 1990
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 3.790431266846362e-05,
+      "loss": 0.0738,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 3.7567385444743934e-05,
+      "loss": 0.0756,
+      "step": 2010
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 3.723045822102426e-05,
+      "loss": 0.0695,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 3.689353099730458e-05,
+      "loss": 0.0653,
+      "step": 2030
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 3.655660377358491e-05,
+      "loss": 0.0701,
+      "step": 2040
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 0.8912965059280396,
+      "learning_rate": 3.621967654986524e-05,
+      "loss": 0.0672,
+      "step": 2050
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 0.9999990463256836,
+      "learning_rate": 3.588274932614555e-05,
+      "loss": 0.069,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 3.554582210242588e-05,
+      "loss": 0.0608,
+      "step": 2070
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 3.52088948787062e-05,
+      "loss": 0.0691,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 3.4871967654986526e-05,
+      "loss": 0.0619,
+      "step": 2090
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 3.453504043126685e-05,
+      "loss": 0.0591,
+      "step": 2100
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 3.419811320754717e-05,
+      "loss": 0.0718,
+      "step": 2110
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 0.999999463558197,
+      "learning_rate": 3.3861185983827494e-05,
+      "loss": 0.0631,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 0.9999996423721313,
+      "learning_rate": 3.3524258760107816e-05,
+      "loss": 0.0691,
+      "step": 2130
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 0.9999990463256836,
+      "learning_rate": 3.3187331536388145e-05,
+      "loss": 0.0647,
+      "step": 2140
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 3.285040431266847e-05,
+      "loss": 0.0583,
+      "step": 2150
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 3.251347708894879e-05,
+      "loss": 0.054,
+      "step": 2160
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 0.9889719486236572,
+      "learning_rate": 3.217654986522911e-05,
+      "loss": 0.0616,
+      "step": 2170
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 0.815100371837616,
+      "learning_rate": 3.1839622641509435e-05,
+      "loss": 0.0658,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 0.92185378074646,
+      "learning_rate": 3.150269541778976e-05,
+      "loss": 0.0603,
+      "step": 2190
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 0.6266804337501526,
+      "learning_rate": 3.1165768194070086e-05,
+      "loss": 0.0559,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 3.08288409703504e-05,
+      "loss": 0.0654,
+      "step": 2210
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 3.0491913746630728e-05,
+      "loss": 0.0579,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 3.0154986522911054e-05,
+      "loss": 0.0572,
+      "step": 2230
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 2.9818059299191376e-05,
+      "loss": 0.0597,
+      "step": 2240
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.9999989867210388,
+      "learning_rate": 2.9481132075471702e-05,
+      "loss": 0.0536,
+      "step": 2250
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 0.8040491938591003,
+      "learning_rate": 2.914420485175202e-05,
+      "loss": 0.0534,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 0.6975265145301819,
+      "learning_rate": 2.8807277628032347e-05,
+      "loss": 0.0536,
+      "step": 2270
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 0.5800132751464844,
+      "learning_rate": 2.847035040431267e-05,
+      "loss": 0.0479,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 2.8133423180592995e-05,
+      "loss": 0.0555,
+      "step": 2290
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 0.9876465201377869,
+      "learning_rate": 2.7796495956873314e-05,
+      "loss": 0.0507,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 0.8521175980567932,
+      "learning_rate": 2.745956873315364e-05,
+      "loss": 0.0548,
+      "step": 2310
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 0.80040043592453,
+      "learning_rate": 2.7122641509433965e-05,
+      "loss": 0.0515,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 0.6687982082366943,
+      "learning_rate": 2.6785714285714288e-05,
+      "loss": 0.0526,
+      "step": 2330
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 0.7264440655708313,
+      "learning_rate": 2.6448787061994614e-05,
+      "loss": 0.0542,
+      "step": 2340
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 0.8285744190216064,
+      "learning_rate": 2.6111859838274933e-05,
+      "loss": 0.0535,
+      "step": 2350
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 2.577493261455526e-05,
+      "loss": 0.0534,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 2.5438005390835577e-05,
+      "loss": 0.0594,
+      "step": 2370
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 0.881329357624054,
+      "learning_rate": 2.5101078167115903e-05,
+      "loss": 0.0542,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 2.476415094339623e-05,
+      "loss": 0.052,
+      "step": 2390
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 0.9246508479118347,
+      "learning_rate": 2.442722371967655e-05,
+      "loss": 0.0564,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 0.909809410572052,
+      "learning_rate": 2.4090296495956874e-05,
+      "loss": 0.0516,
+      "step": 2410
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 2.3753369272237196e-05,
+      "loss": 0.0488,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 0.6667732000350952,
+      "learning_rate": 2.341644204851752e-05,
+      "loss": 0.0484,
+      "step": 2430
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 2.3079514824797844e-05,
+      "loss": 0.0486,
+      "step": 2440
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 0.7443352937698364,
+      "learning_rate": 2.274258760107817e-05,
+      "loss": 0.046,
+      "step": 2450
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 0.9201543927192688,
+      "learning_rate": 2.2405660377358493e-05,
+      "loss": 0.0509,
+      "step": 2460
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 0.9005343914031982,
+      "learning_rate": 2.2068733153638815e-05,
+      "loss": 0.049,
+      "step": 2470
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 0.9772098660469055,
+      "learning_rate": 2.1731805929919137e-05,
+      "loss": 0.0443,
+      "step": 2480
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 0.8524183630943298,
+      "learning_rate": 2.1394878706199463e-05,
+      "loss": 0.0457,
+      "step": 2490
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 0.9903967380523682,
+      "learning_rate": 2.1057951482479785e-05,
+      "loss": 0.0468,
+      "step": 2500
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 0.8543617725372314,
+      "learning_rate": 2.0721024258760108e-05,
+      "loss": 0.0474,
+      "step": 2510
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 0.8450462222099304,
+      "learning_rate": 2.038409703504043e-05,
+      "loss": 0.0447,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 0.7971166968345642,
+      "learning_rate": 2.0047169811320756e-05,
+      "loss": 0.0466,
+      "step": 2530
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 0.8250380754470825,
+      "learning_rate": 1.971024258760108e-05,
+      "loss": 0.0462,
+      "step": 2540
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 1.9373315363881404e-05,
+      "loss": 0.0462,
+      "step": 2550
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 0.9134150147438049,
+      "learning_rate": 1.9036388140161727e-05,
+      "loss": 0.0442,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 1.869946091644205e-05,
+      "loss": 0.0482,
+      "step": 2570
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 1.836253369272237e-05,
+      "loss": 0.0433,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 0.7103074789047241,
+      "learning_rate": 1.8025606469002694e-05,
+      "loss": 0.048,
+      "step": 2590
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 1.768867924528302e-05,
+      "loss": 0.0475,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 0.6229370832443237,
+      "learning_rate": 1.7351752021563345e-05,
+      "loss": 0.0424,
+      "step": 2610
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 0.5841905474662781,
+      "learning_rate": 1.7014824797843668e-05,
+      "loss": 0.0437,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 0.6156032085418701,
+      "learning_rate": 1.667789757412399e-05,
+      "loss": 0.0406,
+      "step": 2630
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 0.741146445274353,
+      "learning_rate": 1.6340970350404313e-05,
+      "loss": 0.0453,
+      "step": 2640
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 0.9999990463256836,
+      "learning_rate": 1.600404312668464e-05,
+      "loss": 0.0446,
+      "step": 2650
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 0.9699227213859558,
+      "learning_rate": 1.566711590296496e-05,
+      "loss": 0.0429,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 1.5330188679245283e-05,
+      "loss": 0.045,
+      "step": 2670
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 0.705869197845459,
+      "learning_rate": 1.4993261455525606e-05,
+      "loss": 0.0385,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 0.5766115188598633,
+      "learning_rate": 1.465633423180593e-05,
+      "loss": 0.0408,
+      "step": 2690
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 0.7847941517829895,
+      "learning_rate": 1.4319407008086256e-05,
+      "loss": 0.0417,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 0.8009430766105652,
+      "learning_rate": 1.3982479784366578e-05,
+      "loss": 0.0454,
+      "step": 2710
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 1.3645552560646902e-05,
+      "loss": 0.0429,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 1.3308625336927224e-05,
+      "loss": 0.0437,
+      "step": 2730
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 0.7371993660926819,
+      "learning_rate": 1.2971698113207547e-05,
+      "loss": 0.0434,
+      "step": 2740
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.6701787710189819,
+      "learning_rate": 1.2634770889487871e-05,
+      "loss": 0.0408,
+      "step": 2750
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 0.9999990463256836,
+      "learning_rate": 1.2297843665768195e-05,
+      "loss": 0.1358,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 0.9999995231628418,
+      "learning_rate": 1.1960916442048519e-05,
+      "loss": 0.0444,
+      "step": 2770
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 0.7984168529510498,
+      "learning_rate": 1.1623989218328842e-05,
+      "loss": 0.039,
+      "step": 2780
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 0.6811168789863586,
+      "learning_rate": 1.1287061994609164e-05,
+      "loss": 0.0412,
+      "step": 2790
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 1.0950134770889488e-05,
+      "loss": 0.0403,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 0.5409925580024719,
+      "learning_rate": 1.0613207547169812e-05,
+      "loss": 0.041,
+      "step": 2810
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 0.616387665271759,
+      "learning_rate": 1.0276280323450135e-05,
+      "loss": 0.0392,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 0.5430307388305664,
+      "learning_rate": 9.939353099730459e-06,
+      "loss": 0.0424,
+      "step": 2830
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 0.6832756400108337,
+      "learning_rate": 9.602425876010781e-06,
+      "loss": 0.043,
+      "step": 2840
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 0.9999991655349731,
+      "learning_rate": 9.265498652291107e-06,
+      "loss": 0.0445,
+      "step": 2850
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 0.8125210404396057,
+      "learning_rate": 8.92857142857143e-06,
+      "loss": 0.0398,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 0.6367383599281311,
+      "learning_rate": 8.591644204851752e-06,
+      "loss": 0.0376,
+      "step": 2870
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 0.5974398851394653,
+      "learning_rate": 8.254716981132076e-06,
+      "loss": 0.0413,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 0.75913405418396,
+      "learning_rate": 7.9177897574124e-06,
+      "loss": 0.0431,
+      "step": 2890
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 0.7355629205703735,
+      "learning_rate": 7.580862533692723e-06,
+      "loss": 0.0466,
+      "step": 2900
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 0.9921271800994873,
+      "learning_rate": 7.243935309973046e-06,
+      "loss": 0.0421,
+      "step": 2910
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 0.908806324005127,
+      "learning_rate": 6.90700808625337e-06,
+      "loss": 0.0398,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 0.9985591173171997,
+      "learning_rate": 6.570080862533692e-06,
+      "loss": 0.0396,
+      "step": 2930
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 0.7728786468505859,
+      "learning_rate": 6.233153638814016e-06,
+      "loss": 0.0409,
+      "step": 2940
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 5.89622641509434e-06,
+      "loss": 0.0393,
+      "step": 2950
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 0.6850334405899048,
+      "learning_rate": 5.5592991913746634e-06,
+      "loss": 0.039,
+      "step": 2960
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 0.6554689407348633,
+      "learning_rate": 5.222371967654987e-06,
+      "loss": 0.044,
+      "step": 2970
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 0.9153503179550171,
+      "learning_rate": 4.88544474393531e-06,
+      "loss": 0.0396,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 0.938971221446991,
+      "learning_rate": 4.548517520215634e-06,
+      "loss": 0.0391,
+      "step": 2990
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 0.7775676846504211,
+      "learning_rate": 4.211590296495957e-06,
+      "loss": 0.0402,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 0.7696932554244995,
+      "learning_rate": 3.8746630727762805e-06,
+      "loss": 0.0389,
+      "step": 3010
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 3.5377358490566038e-06,
+      "loss": 0.0405,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 0.8790382742881775,
+      "learning_rate": 3.200808625336928e-06,
+      "loss": 0.0382,
+      "step": 3030
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 0.9999993443489075,
+      "learning_rate": 2.8638814016172507e-06,
+      "loss": 0.04,
+      "step": 3040
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 0.9999990463256836,
+      "learning_rate": 2.5269541778975744e-06,
+      "loss": 0.0381,
+      "step": 3050
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 0.9999992847442627,
+      "learning_rate": 2.1900269541778976e-06,
+      "loss": 0.043,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 0.6394308805465698,
+      "learning_rate": 1.853099730458221e-06,
+      "loss": 0.0386,
+      "step": 3070
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 0.5383890867233276,
+      "learning_rate": 1.5161725067385445e-06,
+      "loss": 0.0383,
+      "step": 3080
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 0.9999992251396179,
+      "learning_rate": 1.179245283018868e-06,
+      "loss": 0.0394,
+      "step": 3090
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 0.7525647878646851,
+      "learning_rate": 8.423180592991913e-07,
+      "loss": 0.0393,
+      "step": 3100
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 0.9999991059303284,
+      "learning_rate": 5.053908355795148e-07,
+      "loss": 0.0449,
+      "step": 3110
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 0.8186839818954468,
+      "learning_rate": 1.684636118598383e-07,
+      "loss": 0.0356,
+      "step": 3120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 3125,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3125/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:407e581fcd52f78a4810428d6f8dfb6b1abc4db6e6d516ea68f45e62fc2f6520
+size 5304

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "gpt2-medium",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "n_special": 0,
+  "predict_special_tokens": true,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "transformers_version": "4.45.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_logs.csv

@@ -0,0 +1,314 @@
+step,training_loss,grad_norm
+10,22.4391,1.0
+20,18.54,1.0
+30,18.87,1.0
+40,17.984,0.9999998807907104
+50,15.0193,0.9999999403953552
+60,14.2766,0.9999999403953552
+70,11.9355,1.0000001192092896
+80,9.6631,0.9999999403953552
+90,8.2441,0.9999998807907104
+100,7.0992,0.9999998807907104
+110,5.2436,0.9999999403953552
+120,5.0585,0.9999999403953552
+130,4.3211,0.9999999403953552
+140,3.745,0.9999999403953552
+150,3.5158,0.9999998807907104
+160,3.2362,0.9999998807907104
+170,2.8799,0.9999998211860657
+180,2.7697,0.9999999403953552
+190,2.7384,0.9999999403953552
+200,2.4178,0.9999998807907104
+210,2.3849,0.9999998807907104
+220,2.1359,0.9999998807907104
+230,2.2128,0.9999998807907104
+240,2.12,0.9999998211860657
+250,1.9436,0.9999998807907104
+260,1.9452,0.9999998211860657
+270,1.9896,0.9999998211860657
+280,1.8534,0.9999998211860657
+290,1.5842,0.9999999403953552
+300,1.8386,0.9999998807907104
+310,1.7605,0.9999998807907104
+320,1.6376,0.9999998807907104
+330,1.4613,0.9999998807907104
+340,1.5167,0.9999997615814209
+350,1.5203,0.9999998807907104
+360,1.3776,0.9999998211860657
+370,1.3961,0.9999998211860657
+380,1.3971,0.9999998211860657
+390,1.2078,0.9999998211860657
+400,1.1744,0.9999998211860657
+410,1.1205,0.9999997615814209
+420,1.0612,0.9999998807907104
+430,1.0844,0.9999998211860657
+440,1.1055,0.9999998211860657
+450,1.0199,0.9999997019767761
+460,0.987,0.9999998211860657
+470,0.9312,0.9999997615814209
+480,0.9257,0.9999997019767761
+490,0.828,0.9999997615814209
+500,0.9471,0.9999997615814209
+510,0.8311,0.9999998211860657
+520,0.8505,0.9999998211860657
+530,0.7594,0.9999997019767761
+540,0.817,0.9999998807907104
+550,0.7966,0.9999998211860657
+560,0.7625,0.9999998807907104
+570,0.724,0.9999997019767761
+580,0.7257,0.9999997615814209
+590,0.7533,0.9999998211860657
+600,0.707,0.9999997615814209
+610,0.66,0.9999997615814209
+620,0.6193,0.9999998807907104
+630,0.6132,0.9999998807907104
+640,0.6331,0.9999999403953552
+650,0.5495,0.9999998211860657
+660,0.5478,0.9999998211860657
+670,0.5972,0.9999999403953552
+680,0.5714,0.9999997019767761
+690,0.5494,0.9999998211860657
+700,0.5474,0.9999998211860657
+710,0.543,0.9999998211860657
+720,0.5209,0.9999998211860657
+730,0.509,0.9999997615814209
+740,0.555,0.9999998807907104
+750,0.4649,0.9999997615814209
+760,0.4629,0.9999997615814209
+770,0.4924,0.9999997615814209
+780,0.4953,0.9999998211860657
+790,0.507,0.9999997615814209
+800,0.5165,0.9999997615814209
+810,0.4489,0.9999997615814209
+820,0.4279,0.9999998211860657
+830,0.4258,0.9999999403953552
+840,0.4614,0.9999997615814209
+850,0.4691,0.9999998807907104
+860,0.4586,0.9999997615814209
+870,0.4347,0.9999998211860657
+880,0.4125,0.9999997615814209
+890,0.3699,0.9999997019767761
+900,0.441,0.9999998211860657
+910,0.4196,0.9999997019767761
+920,0.36,0.9999997615814209
+930,0.3673,0.9999997019767761
+940,0.3624,0.9999998211860657
+950,0.4013,0.9999997615814209
+960,0.3394,0.9999997019767761
+970,0.3839,0.9999997019767761
+980,0.3663,0.9999998211860657
+990,0.3547,0.9999997615814209
+1000,0.377,0.9999997615814209
+1010,0.3347,0.9999995827674866
+1020,0.3758,0.9999998807907104
+1030,0.3692,0.9999995827674866
+1040,0.3215,0.9999997019767761
+1050,0.3417,0.9999997019767761
+1060,0.3101,0.9999998211860657
+1070,0.3272,0.9999999403953552
+1080,0.368,0.9999999403953552
+1090,0.3303,0.9999995827674866
+1100,0.2986,0.9999995231628418
+1110,0.2911,0.9999994039535522
+1120,0.3068,0.9999997615814209
+1130,0.268,0.9999998211860657
+1140,0.2994,0.9999996423721313
+1150,0.2921,0.9999998211860657
+1160,0.2631,0.9999998211860657
+1170,0.2519,0.9999996423721313
+1180,0.2571,0.9999997615814209
+1190,0.2621,0.9999999403953552
+1200,0.2332,0.9999997615814209
+1210,0.2531,0.9999997019767761
+1220,0.2334,0.9999997019767761
+1230,0.2208,0.9999995827674866
+1240,0.234,0.9999996423721313
+1250,0.2321,0.9999995231628418
+1260,0.2057,0.9999996423721313
+1270,0.2129,0.9999994039535522
+1280,0.2195,0.9999997019767761
+1290,0.2309,0.9999997019767761
+1300,0.1961,0.999999463558197
+1310,0.2059,0.9999995827674866
+1320,0.1896,0.9999992847442627
+1330,0.1952,0.9999997615814209
+1340,0.1865,0.9999994039535522
+1350,0.1781,0.9999995827674866
+1360,0.1965,0.999999463558197
+1370,0.1563,0.9999993443489075
+1380,0.1561,0.9999995231628418
+1390,0.1601,0.9999995827674866
+1400,0.1607,0.999999463558197
+1410,0.1637,0.9999996423721313
+1420,0.1591,0.999999463558197
+1430,0.1782,0.9999993443489075
+1440,0.1434,0.9999996423721313
+1450,0.1356,0.9999993443489075
+1460,0.1321,0.9999993443489075
+1470,0.1417,0.9999994039535522
+1480,0.1444,0.999999463558197
+1490,0.1284,0.999999463558197
+1500,0.1241,0.9999996423721313
+1510,0.1312,0.9999995827674866
+1520,0.1299,0.999999463558197
+1530,0.1332,0.9999991655349731
+1540,0.1232,0.9999991059303284
+1550,0.1147,0.9999993443489075
+1560,0.1284,0.9999997019767761
+1570,0.1233,0.9999995231628418
+1580,0.1134,0.9999992847442627
+1590,0.1194,0.9999996423721313
+1600,0.1188,0.999999463558197
+1610,0.1333,0.9999993443489075
+1620,0.143,0.999999463558197
+1630,0.1038,0.9999992847442627
+1640,0.1689,0.9999993443489075
+1650,0.1065,0.9999992847442627
+1660,0.1033,0.9999995827674866
+1670,0.1248,0.9999991059303284
+1680,0.1068,0.9999995231628418
+1690,0.0982,0.9999991655349731
+1700,0.0901,0.999999463558197
+1710,0.0866,0.9999992847442627
+1720,0.0891,0.9999991655349731
+1730,0.0946,0.9999991655349731
+1740,0.1158,0.9999995231628418
+1750,0.1039,0.9999992251396179
+1760,0.0891,0.810341477394104
+1770,0.0821,0.9999993443489075
+1780,0.0895,0.9601090550422668
+1790,0.0987,0.999999463558197
+1800,0.0932,0.9999993443489075
+1810,0.084,0.9999991655349731
+1820,0.0772,0.9999991655349731
+1830,0.08,0.9272804856300354
+1840,0.0849,0.9999995231628418
+1850,0.0845,0.9999990463256836
+1860,0.0706,0.9999992847442627
+1870,0.0763,0.9999993443489075
+1880,0.0847,0.9999996423721313
+1890,0.0753,0.9999995827674866
+1900,0.0799,0.9999995231628418
+1910,0.0799,0.9999995231628418
+1920,0.0799,0.9999997019767761
+1930,0.076,0.9999989867210388
+1940,0.0669,0.9999992251396179
+1950,0.0699,0.7363486289978027
+1960,0.0716,0.9999992251396179
+1970,0.0724,0.9999989867210388
+1980,0.0717,0.9999994039535522
+1990,0.0669,0.999999463558197
+2000,0.0738,0.9999992251396179
+2010,0.0756,0.9999992251396179
+2020,0.0695,0.9999992847442627
+2030,0.0653,0.9999992847442627
+2040,0.0701,0.9999991059303284
+2050,0.0672,0.8912965059280396
+2060,0.069,0.9999990463256836
+2070,0.0608,0.9999992847442627
+2080,0.0691,0.9999993443489075
+2090,0.0619,0.9999992251396179
+2100,0.0591,0.9999991059303284
+2110,0.0718,0.9999991059303284
+2120,0.0631,0.999999463558197
+2130,0.0691,0.9999996423721313
+2140,0.0647,0.9999990463256836
+2150,0.0583,0.9999991655349731
+2160,0.054,0.9999993443489075
+2170,0.0616,0.9889719486236572
+2180,0.0658,0.815100371837616
+2190,0.0603,0.92185378074646
+2200,0.0559,0.6266804337501526
+2210,0.0654,0.9999993443489075
+2220,0.0579,0.9999991655349731
+2230,0.0572,0.9999991059303284
+2240,0.0597,0.9999991655349731
+2250,0.0536,0.9999989867210388
+2260,0.0534,0.8040491938591003
+2270,0.0536,0.6975265145301819
+2280,0.0479,0.5800132751464844
+2290,0.0555,0.9999991655349731
+2300,0.0507,0.9876465201377869
+2310,0.0548,0.8521175980567932
+2320,0.0515,0.80040043592453
+2330,0.0526,0.6687982082366943
+2340,0.0542,0.7264440655708313
+2350,0.0535,0.8285744190216064
+2360,0.0534,0.9999993443489075
+2370,0.0594,0.9999991655349731
+2380,0.0542,0.881329357624054
+2390,0.052,0.9999991655349731
+2400,0.0564,0.9246508479118347
+2410,0.0516,0.909809410572052
+2420,0.0488,0.9999992251396179
+2430,0.0484,0.6667732000350952
+2440,0.0486,0.9999993443489075
+2450,0.046,0.7443352937698364
+2460,0.0509,0.9201543927192688
+2470,0.049,0.9005343914031982
+2480,0.0443,0.9772098660469055
+2490,0.0457,0.8524183630943298
+2500,0.0468,0.9903967380523682
+2510,0.0474,0.8543617725372314
+2520,0.0447,0.8450462222099304
+2530,0.0466,0.7971166968345642
+2540,0.0462,0.8250380754470825
+2550,0.0462,0.9999992847442627
+2560,0.0442,0.9134150147438049
+2570,0.0482,0.9999995231628418
+2580,0.0433,0.9999993443489075
+2590,0.048,0.7103074789047241
+2600,0.0475,0.9999991059303284
+2610,0.0424,0.6229370832443237
+2620,0.0437,0.5841905474662781
+2630,0.0406,0.6156032085418701
+2640,0.0453,0.741146445274353
+2650,0.0446,0.9999990463256836
+2660,0.0429,0.9699227213859558
+2670,0.045,0.9999992251396179
+2680,0.0385,0.705869197845459
+2690,0.0408,0.5766115188598633
+2700,0.0417,0.7847941517829895
+2710,0.0454,0.8009430766105652
+2720,0.0429,0.9999991059303284
+2730,0.0437,0.9999992251396179
+2740,0.0434,0.7371993660926819
+2750,0.0408,0.6701787710189819
+2760,0.1358,0.9999990463256836
+2770,0.0444,0.9999995231628418
+2780,0.039,0.7984168529510498
+2790,0.0412,0.6811168789863586
+2800,0.0403,0.9999991655349731
+2810,0.041,0.5409925580024719
+2820,0.0392,0.616387665271759
+2830,0.0424,0.5430307388305664
+2840,0.043,0.6832756400108337
+2850,0.0445,0.9999991655349731
+2860,0.0398,0.8125210404396057
+2870,0.0376,0.6367383599281311
+2880,0.0413,0.5974398851394653
+2890,0.0431,0.75913405418396
+2900,0.0466,0.7355629205703735
+2910,0.0421,0.9921271800994873
+2920,0.0398,0.908806324005127
+2930,0.0396,0.9985591173171997
+2940,0.0409,0.7728786468505859
+2950,0.0393,0.9999991059303284
+2960,0.039,0.6850334405899048
+2970,0.044,0.6554689407348633
+2980,0.0396,0.9153503179550171
+2990,0.0391,0.938971221446991
+3000,0.0402,0.7775676846504211
+3010,0.0389,0.7696932554244995
+3020,0.0405,0.9999991059303284
+3030,0.0382,0.8790382742881775
+3040,0.04,0.9999993443489075
+3050,0.0381,0.9999990463256836
+3060,0.043,0.9999992847442627
+3070,0.0386,0.6394308805465698
+3080,0.0383,0.5383890867233276
+3090,0.0394,0.9999992251396179
+3100,0.0393,0.7525647878646851
+3110,0.0449,0.9999991059303284
+3120,0.0356,0.8186839818954468
+3125,nan,nan