Upload folder using huggingface_hub

README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9068919e58b963cc529477ba8fd3da050adb2872365aa0e549cf2568840369aa
+size 25172088

checkpoint-3125/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

checkpoint-3125/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-3125/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9068919e58b963cc529477ba8fd3da050adb2872365aa0e549cf2568840369aa
+size 25172088

checkpoint-3125/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e65f6f65a42fbe2fb3c2a25fc3e4a9282dd96fca194f2d99cf2d6c6668ae9b28
+size 50372538

checkpoint-3125/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5994634e90ce1b45d012a45568063be05fea876e791cd66b48a4efc924164b2
+size 14244

checkpoint-3125/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e42e57efa8b95889e87232fe6ea0805483ed6c8a2d35bb60230715b624ff5bda
+size 1064

checkpoint-3125/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-3125/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-3125/trainer_state.json

@@ -0,0 +1,2217 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 3125,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 5.000000476837158,
+      "learning_rate": 6.369426751592357e-06,
+      "loss": 22.4391,
+      "step": 10
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 5.000000476837158,
+      "learning_rate": 1.2738853503184714e-05,
+      "loss": 18.5406,
+      "step": 20
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 5.0,
+      "learning_rate": 1.910828025477707e-05,
+      "loss": 18.8756,
+      "step": 30
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 2.5477707006369428e-05,
+      "loss": 18.0104,
+      "step": 40
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 3.184713375796178e-05,
+      "loss": 15.1106,
+      "step": 50
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 3.821656050955414e-05,
+      "loss": 14.5177,
+      "step": 60
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 5.0,
+      "learning_rate": 4.45859872611465e-05,
+      "loss": 12.4196,
+      "step": 70
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 5.0,
+      "learning_rate": 5.0955414012738855e-05,
+      "loss": 10.4176,
+      "step": 80
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 5.732484076433121e-05,
+      "loss": 9.3694,
+      "step": 90
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 6.369426751592356e-05,
+      "loss": 8.6032,
+      "step": 100
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.006369426751592e-05,
+      "loss": 6.5879,
+      "step": 110
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.643312101910829e-05,
+      "loss": 6.7468,
+      "step": 120
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.280254777070065e-05,
+      "loss": 5.6174,
+      "step": 130
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 5.0,
+      "learning_rate": 8.9171974522293e-05,
+      "loss": 5.1907,
+      "step": 140
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.554140127388536e-05,
+      "loss": 4.9405,
+      "step": 150
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.989892183288411e-05,
+      "loss": 4.4357,
+      "step": 160
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 5.0,
+      "learning_rate": 9.956199460916442e-05,
+      "loss": 4.1208,
+      "step": 170
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.922506738544474e-05,
+      "loss": 4.103,
+      "step": 180
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.888814016172507e-05,
+      "loss": 4.0038,
+      "step": 190
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 5.0,
+      "learning_rate": 9.85512129380054e-05,
+      "loss": 3.4375,
+      "step": 200
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.821428571428572e-05,
+      "loss": 3.4134,
+      "step": 210
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 5.0,
+      "learning_rate": 9.787735849056603e-05,
+      "loss": 3.1147,
+      "step": 220
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.754043126684636e-05,
+      "loss": 3.2726,
+      "step": 230
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.720350404312669e-05,
+      "loss": 3.0596,
+      "step": 240
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.686657681940702e-05,
+      "loss": 2.9703,
+      "step": 250
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.652964959568734e-05,
+      "loss": 2.9099,
+      "step": 260
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.619272237196765e-05,
+      "loss": 3.084,
+      "step": 270
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.585579514824798e-05,
+      "loss": 2.7716,
+      "step": 280
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.551886792452831e-05,
+      "loss": 2.5286,
+      "step": 290
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.518194070080863e-05,
+      "loss": 2.9801,
+      "step": 300
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.484501347708896e-05,
+      "loss": 2.6343,
+      "step": 310
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.450808625336927e-05,
+      "loss": 2.5548,
+      "step": 320
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.41711590296496e-05,
+      "loss": 2.3042,
+      "step": 330
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.383423180592993e-05,
+      "loss": 2.1663,
+      "step": 340
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 5.0,
+      "learning_rate": 9.349730458221025e-05,
+      "loss": 2.2429,
+      "step": 350
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.316037735849057e-05,
+      "loss": 1.9972,
+      "step": 360
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.282345013477089e-05,
+      "loss": 1.9431,
+      "step": 370
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 5.0,
+      "learning_rate": 9.248652291105122e-05,
+      "loss": 1.9534,
+      "step": 380
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 9.214959568733154e-05,
+      "loss": 1.7552,
+      "step": 390
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 5.0,
+      "learning_rate": 9.181266846361186e-05,
+      "loss": 1.7837,
+      "step": 400
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 5.0,
+      "learning_rate": 9.14757412398922e-05,
+      "loss": 1.6729,
+      "step": 410
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 4.999998569488525,
+      "learning_rate": 9.113881401617251e-05,
+      "loss": 1.5536,
+      "step": 420
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.080188679245284e-05,
+      "loss": 1.6502,
+      "step": 430
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 5.0,
+      "learning_rate": 9.046495956873315e-05,
+      "loss": 1.4372,
+      "step": 440
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 9.012803234501348e-05,
+      "loss": 1.43,
+      "step": 450
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.979110512129381e-05,
+      "loss": 1.4757,
+      "step": 460
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.945417789757413e-05,
+      "loss": 1.323,
+      "step": 470
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 4.999998569488525,
+      "learning_rate": 8.911725067385444e-05,
+      "loss": 1.257,
+      "step": 480
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.878032345013477e-05,
+      "loss": 1.2685,
+      "step": 490
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.84433962264151e-05,
+      "loss": 1.3465,
+      "step": 500
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.810646900269543e-05,
+      "loss": 1.1891,
+      "step": 510
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.776954177897575e-05,
+      "loss": 1.271,
+      "step": 520
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.743261455525606e-05,
+      "loss": 1.1265,
+      "step": 530
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.709568733153639e-05,
+      "loss": 1.2308,
+      "step": 540
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.675876010781672e-05,
+      "loss": 1.1909,
+      "step": 550
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 4.743152618408203,
+      "learning_rate": 8.642183288409704e-05,
+      "loss": 1.1423,
+      "step": 560
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.608490566037735e-05,
+      "loss": 1.1738,
+      "step": 570
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.574797843665768e-05,
+      "loss": 1.1288,
+      "step": 580
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.541105121293801e-05,
+      "loss": 1.0923,
+      "step": 590
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.507412398921834e-05,
+      "loss": 1.0964,
+      "step": 600
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.473719676549866e-05,
+      "loss": 0.9797,
+      "step": 610
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 4.760279655456543,
+      "learning_rate": 8.440026954177897e-05,
+      "loss": 1.0295,
+      "step": 620
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.40633423180593e-05,
+      "loss": 0.9473,
+      "step": 630
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.372641509433963e-05,
+      "loss": 1.0278,
+      "step": 640
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 5.0,
+      "learning_rate": 8.338948787061996e-05,
+      "loss": 0.9588,
+      "step": 650
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.305256064690027e-05,
+      "loss": 0.954,
+      "step": 660
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.271563342318059e-05,
+      "loss": 0.9665,
+      "step": 670
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.237870619946092e-05,
+      "loss": 0.9109,
+      "step": 680
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.204177897574125e-05,
+      "loss": 0.9,
+      "step": 690
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.170485175202158e-05,
+      "loss": 0.8534,
+      "step": 700
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.136792452830189e-05,
+      "loss": 0.8176,
+      "step": 710
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.103099730458221e-05,
+      "loss": 0.8267,
+      "step": 720
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 4.478681564331055,
+      "learning_rate": 8.069407008086254e-05,
+      "loss": 0.8188,
+      "step": 730
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 8.035714285714287e-05,
+      "loss": 0.8477,
+      "step": 740
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 8.002021563342318e-05,
+      "loss": 0.789,
+      "step": 750
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.968328840970351e-05,
+      "loss": 0.7381,
+      "step": 760
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.934636118598383e-05,
+      "loss": 0.754,
+      "step": 770
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.900943396226416e-05,
+      "loss": 0.7388,
+      "step": 780
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 5.0,
+      "learning_rate": 7.867250673854449e-05,
+      "loss": 0.8036,
+      "step": 790
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.83355795148248e-05,
+      "loss": 0.803,
+      "step": 800
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.799865229110512e-05,
+      "loss": 0.7286,
+      "step": 810
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.766172506738545e-05,
+      "loss": 0.6898,
+      "step": 820
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.732479784366577e-05,
+      "loss": 0.6662,
+      "step": 830
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.69878706199461e-05,
+      "loss": 0.6753,
+      "step": 840
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.665094339622642e-05,
+      "loss": 0.6469,
+      "step": 850
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.631401617250674e-05,
+      "loss": 0.6496,
+      "step": 860
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.597708894878706e-05,
+      "loss": 0.6454,
+      "step": 870
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 4.928877830505371,
+      "learning_rate": 7.56401617250674e-05,
+      "loss": 0.5762,
+      "step": 880
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.530323450134771e-05,
+      "loss": 0.6006,
+      "step": 890
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.496630727762804e-05,
+      "loss": 0.6075,
+      "step": 900
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 4.305392742156982,
+      "learning_rate": 7.462938005390835e-05,
+      "loss": 0.5837,
+      "step": 910
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.429245283018868e-05,
+      "loss": 0.5447,
+      "step": 920
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 5.0,
+      "learning_rate": 7.395552560646901e-05,
+      "loss": 0.5516,
+      "step": 930
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 4.88629150390625,
+      "learning_rate": 7.361859838274933e-05,
+      "loss": 0.5024,
+      "step": 940
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 4.116428375244141,
+      "learning_rate": 7.328167115902966e-05,
+      "loss": 0.571,
+      "step": 950
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 4.824604034423828,
+      "learning_rate": 7.294474393530997e-05,
+      "loss": 0.4532,
+      "step": 960
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 3.117197275161743,
+      "learning_rate": 7.26078167115903e-05,
+      "loss": 0.4883,
+      "step": 970
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.227088948787062e-05,
+      "loss": 0.4787,
+      "step": 980
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 4.999999523162842,
+      "learning_rate": 7.193396226415095e-05,
+      "loss": 0.4637,
+      "step": 990
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.159703504043128e-05,
+      "loss": 0.4781,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.126010781671159e-05,
+      "loss": 0.4286,
+      "step": 1010
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 7.092318059299192e-05,
+      "loss": 0.4412,
+      "step": 1020
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 3.544337749481201,
+      "learning_rate": 7.058625336927224e-05,
+      "loss": 0.4406,
+      "step": 1030
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 4.349572658538818,
+      "learning_rate": 7.024932614555257e-05,
+      "loss": 0.4078,
+      "step": 1040
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 4.470551013946533,
+      "learning_rate": 6.99123989218329e-05,
+      "loss": 0.4726,
+      "step": 1050
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 4.85098934173584,
+      "learning_rate": 6.957547169811321e-05,
+      "loss": 0.4033,
+      "step": 1060
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 6.923854447439353e-05,
+      "loss": 0.3966,
+      "step": 1070
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 3.946486711502075,
+      "learning_rate": 6.890161725067386e-05,
+      "loss": 0.4374,
+      "step": 1080
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 3.7488667964935303,
+      "learning_rate": 6.856469002695418e-05,
+      "loss": 0.3775,
+      "step": 1090
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 3.612640857696533,
+      "learning_rate": 6.822776280323451e-05,
+      "loss": 0.3846,
+      "step": 1100
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 6.789083557951483e-05,
+      "loss": 0.3444,
+      "step": 1110
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 4.179413795471191,
+      "learning_rate": 6.755390835579514e-05,
+      "loss": 0.3635,
+      "step": 1120
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 4.016063213348389,
+      "learning_rate": 6.721698113207547e-05,
+      "loss": 0.3313,
+      "step": 1130
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 3.057584762573242,
+      "learning_rate": 6.68800539083558e-05,
+      "loss": 0.3312,
+      "step": 1140
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 2.5960428714752197,
+      "learning_rate": 6.654312668463612e-05,
+      "loss": 0.3141,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 3.4362008571624756,
+      "learning_rate": 6.620619946091643e-05,
+      "loss": 0.3122,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 4.949530601501465,
+      "learning_rate": 6.586927223719676e-05,
+      "loss": 0.3085,
+      "step": 1170
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 3.274339199066162,
+      "learning_rate": 6.553234501347709e-05,
+      "loss": 0.3028,
+      "step": 1180
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 4.999999046325684,
+      "learning_rate": 6.519541778975742e-05,
+      "loss": 0.3128,
+      "step": 1190
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 3.0627970695495605,
+      "learning_rate": 6.485849056603774e-05,
+      "loss": 0.253,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 3.8672235012054443,
+      "learning_rate": 6.452156334231805e-05,
+      "loss": 0.2982,
+      "step": 1210
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 3.373741388320923,
+      "learning_rate": 6.418463611859838e-05,
+      "loss": 0.2763,
+      "step": 1220
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 3.5057952404022217,
+      "learning_rate": 6.384770889487871e-05,
+      "loss": 0.2833,
+      "step": 1230
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 3.149587631225586,
+      "learning_rate": 6.351078167115904e-05,
+      "loss": 0.284,
+      "step": 1240
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 3.716684103012085,
+      "learning_rate": 6.317385444743936e-05,
+      "loss": 0.2553,
+      "step": 1250
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 2.4263858795166016,
+      "learning_rate": 6.283692722371967e-05,
+      "loss": 0.2414,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 3.104553699493408,
+      "learning_rate": 6.25e-05,
+      "loss": 0.2542,
+      "step": 1270
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 2.915567636489868,
+      "learning_rate": 6.216307277628033e-05,
+      "loss": 0.2442,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 3.980184316635132,
+      "learning_rate": 6.182614555256066e-05,
+      "loss": 0.2833,
+      "step": 1290
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 4.143345355987549,
+      "learning_rate": 6.148921832884098e-05,
+      "loss": 0.248,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 1.8089429140090942,
+      "learning_rate": 6.115229110512129e-05,
+      "loss": 0.2426,
+      "step": 1310
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 1.8527010679244995,
+      "learning_rate": 6.081536388140162e-05,
+      "loss": 0.246,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 3.4966213703155518,
+      "learning_rate": 6.047843665768195e-05,
+      "loss": 0.2405,
+      "step": 1330
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 2.331099510192871,
+      "learning_rate": 6.0141509433962265e-05,
+      "loss": 0.2713,
+      "step": 1340
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 3.4839956760406494,
+      "learning_rate": 5.980458221024259e-05,
+      "loss": 0.2202,
+      "step": 1350
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 2.3249175548553467,
+      "learning_rate": 5.9467654986522916e-05,
+      "loss": 0.246,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 3.72847318649292,
+      "learning_rate": 5.913072776280324e-05,
+      "loss": 0.2126,
+      "step": 1370
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 2.6438794136047363,
+      "learning_rate": 5.879380053908357e-05,
+      "loss": 0.2059,
+      "step": 1380
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 3.9793527126312256,
+      "learning_rate": 5.8456873315363884e-05,
+      "loss": 0.2061,
+      "step": 1390
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 3.324707508087158,
+      "learning_rate": 5.8119946091644206e-05,
+      "loss": 0.2063,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 3.0844309329986572,
+      "learning_rate": 5.7783018867924535e-05,
+      "loss": 0.2234,
+      "step": 1410
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 3.1997532844543457,
+      "learning_rate": 5.744609164420486e-05,
+      "loss": 0.2032,
+      "step": 1420
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 2.557157278060913,
+      "learning_rate": 5.710916442048517e-05,
+      "loss": 0.2186,
+      "step": 1430
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 2.0848405361175537,
+      "learning_rate": 5.6772237196765496e-05,
+      "loss": 0.1934,
+      "step": 1440
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 1.8372052907943726,
+      "learning_rate": 5.6435309973045825e-05,
+      "loss": 0.191,
+      "step": 1450
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 2.1945323944091797,
+      "learning_rate": 5.609838274932615e-05,
+      "loss": 0.1689,
+      "step": 1460
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 3.432448625564575,
+      "learning_rate": 5.5761455525606476e-05,
+      "loss": 0.1901,
+      "step": 1470
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 2.2553012371063232,
+      "learning_rate": 5.542452830188679e-05,
+      "loss": 0.2023,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 2.1100335121154785,
+      "learning_rate": 5.5087601078167114e-05,
+      "loss": 0.1777,
+      "step": 1490
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 1.9357300996780396,
+      "learning_rate": 5.4750673854447444e-05,
+      "loss": 0.1672,
+      "step": 1500
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 2.9728143215179443,
+      "learning_rate": 5.4413746630727766e-05,
+      "loss": 0.1865,
+      "step": 1510
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 3.1837971210479736,
+      "learning_rate": 5.407681940700808e-05,
+      "loss": 0.1942,
+      "step": 1520
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 3.62715220451355,
+      "learning_rate": 5.373989218328841e-05,
+      "loss": 0.189,
+      "step": 1530
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 2.986555576324463,
+      "learning_rate": 5.340296495956873e-05,
+      "loss": 0.1781,
+      "step": 1540
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 1.4352009296417236,
+      "learning_rate": 5.306603773584906e-05,
+      "loss": 0.1618,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 2.6908838748931885,
+      "learning_rate": 5.2729110512129385e-05,
+      "loss": 0.1739,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 2.1369287967681885,
+      "learning_rate": 5.23921832884097e-05,
+      "loss": 0.1782,
+      "step": 1570
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 1.9476417303085327,
+      "learning_rate": 5.205525606469003e-05,
+      "loss": 0.1669,
+      "step": 1580
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 2.5264928340911865,
+      "learning_rate": 5.171832884097035e-05,
+      "loss": 0.1634,
+      "step": 1590
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 2.2296202182769775,
+      "learning_rate": 5.138140161725068e-05,
+      "loss": 0.1763,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 1.9065362215042114,
+      "learning_rate": 5.1044474393531e-05,
+      "loss": 0.1863,
+      "step": 1610
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 2.1693010330200195,
+      "learning_rate": 5.070754716981132e-05,
+      "loss": 0.1745,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 1.8745787143707275,
+      "learning_rate": 5.037061994609165e-05,
+      "loss": 0.1465,
+      "step": 1630
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 2.0987377166748047,
+      "learning_rate": 5.003369272237197e-05,
+      "loss": 0.1525,
+      "step": 1640
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 2.405745506286621,
+      "learning_rate": 4.969676549865229e-05,
+      "loss": 0.1525,
+      "step": 1650
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 3.3560268878936768,
+      "learning_rate": 4.9359838274932616e-05,
+      "loss": 0.1447,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 1.3180267810821533,
+      "learning_rate": 4.902291105121294e-05,
+      "loss": 0.1689,
+      "step": 1670
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 1.4368494749069214,
+      "learning_rate": 4.868598382749327e-05,
+      "loss": 0.1489,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 1.5566891431808472,
+      "learning_rate": 4.834905660377358e-05,
+      "loss": 0.1368,
+      "step": 1690
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 2.221500873565674,
+      "learning_rate": 4.801212938005391e-05,
+      "loss": 0.1445,
+      "step": 1700
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 2.9313652515411377,
+      "learning_rate": 4.7675202156334234e-05,
+      "loss": 0.1426,
+      "step": 1710
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 1.836387038230896,
+      "learning_rate": 4.733827493261456e-05,
+      "loss": 0.138,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 1.5651477575302124,
+      "learning_rate": 4.7001347708894886e-05,
+      "loss": 0.1449,
+      "step": 1730
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 3.2158570289611816,
+      "learning_rate": 4.66644204851752e-05,
+      "loss": 0.1438,
+      "step": 1740
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 1.5862022638320923,
+      "learning_rate": 4.632749326145553e-05,
+      "loss": 0.1423,
+      "step": 1750
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 1.6251810789108276,
+      "learning_rate": 4.5990566037735846e-05,
+      "loss": 0.1469,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 1.594459056854248,
+      "learning_rate": 4.5653638814016176e-05,
+      "loss": 0.13,
+      "step": 1770
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 1.554364562034607,
+      "learning_rate": 4.53167115902965e-05,
+      "loss": 0.1309,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 2.1495261192321777,
+      "learning_rate": 4.497978436657682e-05,
+      "loss": 0.1504,
+      "step": 1790
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 2.061843156814575,
+      "learning_rate": 4.464285714285715e-05,
+      "loss": 0.1499,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 1.517041802406311,
+      "learning_rate": 4.4305929919137465e-05,
+      "loss": 0.1393,
+      "step": 1810
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 1.455052137374878,
+      "learning_rate": 4.3969002695417794e-05,
+      "loss": 0.1278,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 2.3193039894104004,
+      "learning_rate": 4.363207547169812e-05,
+      "loss": 0.141,
+      "step": 1830
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 1.6179640293121338,
+      "learning_rate": 4.329514824797844e-05,
+      "loss": 0.1361,
+      "step": 1840
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 2.4831204414367676,
+      "learning_rate": 4.295822102425876e-05,
+      "loss": 0.1333,
+      "step": 1850
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 1.112313985824585,
+      "learning_rate": 4.2621293800539084e-05,
+      "loss": 0.1195,
+      "step": 1860
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 1.4801064729690552,
+      "learning_rate": 4.2284366576819406e-05,
+      "loss": 0.1232,
+      "step": 1870
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 2.4858882427215576,
+      "learning_rate": 4.1947439353099736e-05,
+      "loss": 0.1389,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 1.9607441425323486,
+      "learning_rate": 4.161051212938006e-05,
+      "loss": 0.1236,
+      "step": 1890
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 2.4481492042541504,
+      "learning_rate": 4.127358490566038e-05,
+      "loss": 0.1297,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 1.8709760904312134,
+      "learning_rate": 4.09366576819407e-05,
+      "loss": 0.134,
+      "step": 1910
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 3.084176778793335,
+      "learning_rate": 4.0599730458221025e-05,
+      "loss": 0.1259,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 1.546714186668396,
+      "learning_rate": 4.026280323450135e-05,
+      "loss": 0.1157,
+      "step": 1930
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 2.207704782485962,
+      "learning_rate": 3.992587601078167e-05,
+      "loss": 0.1185,
+      "step": 1940
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 1.6168131828308105,
+      "learning_rate": 3.9588948787062e-05,
+      "loss": 0.1222,
+      "step": 1950
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 3.353578567504883,
+      "learning_rate": 3.9252021563342315e-05,
+      "loss": 0.1156,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 2.5373737812042236,
+      "learning_rate": 3.8915094339622644e-05,
+      "loss": 0.1218,
+      "step": 1970
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 4.4312968254089355,
+      "learning_rate": 3.8578167115902966e-05,
+      "loss": 0.1362,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 1.7041658163070679,
+      "learning_rate": 3.824123989218329e-05,
+      "loss": 0.1139,
+      "step": 1990
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 2.0976693630218506,
+      "learning_rate": 3.790431266846362e-05,
+      "loss": 0.1314,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 1.9576454162597656,
+      "learning_rate": 3.7567385444743934e-05,
+      "loss": 0.1351,
+      "step": 2010
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 1.4846729040145874,
+      "learning_rate": 3.723045822102426e-05,
+      "loss": 0.1123,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 1.9108271598815918,
+      "learning_rate": 3.689353099730458e-05,
+      "loss": 0.1168,
+      "step": 2030
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 1.186391830444336,
+      "learning_rate": 3.655660377358491e-05,
+      "loss": 0.1225,
+      "step": 2040
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 2.4576640129089355,
+      "learning_rate": 3.621967654986524e-05,
+      "loss": 0.1226,
+      "step": 2050
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 1.863196849822998,
+      "learning_rate": 3.588274932614555e-05,
+      "loss": 0.1242,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 1.3311208486557007,
+      "learning_rate": 3.554582210242588e-05,
+      "loss": 0.1098,
+      "step": 2070
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 1.780318021774292,
+      "learning_rate": 3.52088948787062e-05,
+      "loss": 0.1199,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 2.5270535945892334,
+      "learning_rate": 3.4871967654986526e-05,
+      "loss": 0.1051,
+      "step": 2090
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 1.394476294517517,
+      "learning_rate": 3.453504043126685e-05,
+      "loss": 0.1034,
+      "step": 2100
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 1.3372985124588013,
+      "learning_rate": 3.419811320754717e-05,
+      "loss": 0.1265,
+      "step": 2110
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 2.175616979598999,
+      "learning_rate": 3.3861185983827494e-05,
+      "loss": 0.1038,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 2.5146851539611816,
+      "learning_rate": 3.3524258760107816e-05,
+      "loss": 0.1243,
+      "step": 2130
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 1.9544363021850586,
+      "learning_rate": 3.3187331536388145e-05,
+      "loss": 0.1136,
+      "step": 2140
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 1.6316698789596558,
+      "learning_rate": 3.285040431266847e-05,
+      "loss": 0.1087,
+      "step": 2150
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 1.7699568271636963,
+      "learning_rate": 3.251347708894879e-05,
+      "loss": 0.103,
+      "step": 2160
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 1.3398338556289673,
+      "learning_rate": 3.217654986522911e-05,
+      "loss": 0.1147,
+      "step": 2170
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 1.3555325269699097,
+      "learning_rate": 3.1839622641509435e-05,
+      "loss": 0.1093,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 1.2930587530136108,
+      "learning_rate": 3.150269541778976e-05,
+      "loss": 0.1083,
+      "step": 2190
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 1.017906665802002,
+      "learning_rate": 3.1165768194070086e-05,
+      "loss": 0.0958,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 2.231780529022217,
+      "learning_rate": 3.08288409703504e-05,
+      "loss": 0.1284,
+      "step": 2210
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 1.7175219058990479,
+      "learning_rate": 3.0491913746630728e-05,
+      "loss": 0.1039,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 1.588964581489563,
+      "learning_rate": 3.0154986522911054e-05,
+      "loss": 0.1059,
+      "step": 2230
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 1.4871183633804321,
+      "learning_rate": 2.9818059299191376e-05,
+      "loss": 0.1012,
+      "step": 2240
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 1.933687686920166,
+      "learning_rate": 2.9481132075471702e-05,
+      "loss": 0.0946,
+      "step": 2250
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 1.170185923576355,
+      "learning_rate": 2.914420485175202e-05,
+      "loss": 0.096,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 1.33351469039917,
+      "learning_rate": 2.8807277628032347e-05,
+      "loss": 0.1001,
+      "step": 2270
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 1.051422119140625,
+      "learning_rate": 2.847035040431267e-05,
+      "loss": 0.0855,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 2.476947069168091,
+      "learning_rate": 2.8133423180592995e-05,
+      "loss": 0.0903,
+      "step": 2290
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 1.5270748138427734,
+      "learning_rate": 2.7796495956873314e-05,
+      "loss": 0.101,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 1.573987364768982,
+      "learning_rate": 2.745956873315364e-05,
+      "loss": 0.0901,
+      "step": 2310
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 1.211846947669983,
+      "learning_rate": 2.7122641509433965e-05,
+      "loss": 0.1053,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 1.0927597284317017,
+      "learning_rate": 2.6785714285714288e-05,
+      "loss": 0.0956,
+      "step": 2330
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 1.5684590339660645,
+      "learning_rate": 2.6448787061994614e-05,
+      "loss": 0.1084,
+      "step": 2340
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 1.7778979539871216,
+      "learning_rate": 2.6111859838274933e-05,
+      "loss": 0.0986,
+      "step": 2350
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 1.2901784181594849,
+      "learning_rate": 2.577493261455526e-05,
+      "loss": 0.0982,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 1.230678915977478,
+      "learning_rate": 2.5438005390835577e-05,
+      "loss": 0.1115,
+      "step": 2370
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 1.37783944606781,
+      "learning_rate": 2.5101078167115903e-05,
+      "loss": 0.0967,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 1.5255314111709595,
+      "learning_rate": 2.476415094339623e-05,
+      "loss": 0.0984,
+      "step": 2390
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 2.0732505321502686,
+      "learning_rate": 2.442722371967655e-05,
+      "loss": 0.0987,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 1.2267639636993408,
+      "learning_rate": 2.4090296495956874e-05,
+      "loss": 0.0944,
+      "step": 2410
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 2.0225038528442383,
+      "learning_rate": 2.3753369272237196e-05,
+      "loss": 0.0923,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 2.311718463897705,
+      "learning_rate": 2.341644204851752e-05,
+      "loss": 0.096,
+      "step": 2430
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 2.0240392684936523,
+      "learning_rate": 2.3079514824797844e-05,
+      "loss": 0.0935,
+      "step": 2440
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 1.575096607208252,
+      "learning_rate": 2.274258760107817e-05,
+      "loss": 0.0886,
+      "step": 2450
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 1.8266900777816772,
+      "learning_rate": 2.2405660377358493e-05,
+      "loss": 0.0971,
+      "step": 2460
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 1.2168657779693604,
+      "learning_rate": 2.2068733153638815e-05,
+      "loss": 0.0897,
+      "step": 2470
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 1.8267066478729248,
+      "learning_rate": 2.1731805929919137e-05,
+      "loss": 0.0942,
+      "step": 2480
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 1.4711161851882935,
+      "learning_rate": 2.1394878706199463e-05,
+      "loss": 0.0921,
+      "step": 2490
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 1.9106966257095337,
+      "learning_rate": 2.1057951482479785e-05,
+      "loss": 0.0875,
+      "step": 2500
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 2.4690868854522705,
+      "learning_rate": 2.0721024258760108e-05,
+      "loss": 0.0929,
+      "step": 2510
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 1.4428863525390625,
+      "learning_rate": 2.038409703504043e-05,
+      "loss": 0.0876,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 1.3941131830215454,
+      "learning_rate": 2.0047169811320756e-05,
+      "loss": 0.0915,
+      "step": 2530
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 1.9686850309371948,
+      "learning_rate": 1.971024258760108e-05,
+      "loss": 0.09,
+      "step": 2540
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 1.6172926425933838,
+      "learning_rate": 1.9373315363881404e-05,
+      "loss": 0.0939,
+      "step": 2550
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 1.93358314037323,
+      "learning_rate": 1.9036388140161727e-05,
+      "loss": 0.0876,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 1.436843752861023,
+      "learning_rate": 1.869946091644205e-05,
+      "loss": 0.1017,
+      "step": 2570
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 1.1019058227539062,
+      "learning_rate": 1.836253369272237e-05,
+      "loss": 0.0884,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 1.6116135120391846,
+      "learning_rate": 1.8025606469002694e-05,
+      "loss": 0.0906,
+      "step": 2590
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 1.0761088132858276,
+      "learning_rate": 1.768867924528302e-05,
+      "loss": 0.0975,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 1.3172565698623657,
+      "learning_rate": 1.7351752021563345e-05,
+      "loss": 0.0856,
+      "step": 2610
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 1.0831719636917114,
+      "learning_rate": 1.7014824797843668e-05,
+      "loss": 0.0859,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 2.260585308074951,
+      "learning_rate": 1.667789757412399e-05,
+      "loss": 0.0865,
+      "step": 2630
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 1.5628427267074585,
+      "learning_rate": 1.6340970350404313e-05,
+      "loss": 0.0959,
+      "step": 2640
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 1.4175759553909302,
+      "learning_rate": 1.600404312668464e-05,
+      "loss": 0.0862,
+      "step": 2650
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 1.6256355047225952,
+      "learning_rate": 1.566711590296496e-05,
+      "loss": 0.0882,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 1.1748156547546387,
+      "learning_rate": 1.5330188679245283e-05,
+      "loss": 0.0939,
+      "step": 2670
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 1.1186449527740479,
+      "learning_rate": 1.4993261455525606e-05,
+      "loss": 0.0822,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 1.2344774007797241,
+      "learning_rate": 1.465633423180593e-05,
+      "loss": 0.0853,
+      "step": 2690
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 1.4567972421646118,
+      "learning_rate": 1.4319407008086256e-05,
+      "loss": 0.0934,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 1.1431821584701538,
+      "learning_rate": 1.3982479784366578e-05,
+      "loss": 0.0901,
+      "step": 2710
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 1.644754409790039,
+      "learning_rate": 1.3645552560646902e-05,
+      "loss": 0.0849,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 0.9929800033569336,
+      "learning_rate": 1.3308625336927224e-05,
+      "loss": 0.0846,
+      "step": 2730
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 1.2569183111190796,
+      "learning_rate": 1.2971698113207547e-05,
+      "loss": 0.0824,
+      "step": 2740
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 1.3616032600402832,
+      "learning_rate": 1.2634770889487871e-05,
+      "loss": 0.0912,
+      "step": 2750
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 1.4828603267669678,
+      "learning_rate": 1.2297843665768195e-05,
+      "loss": 0.1324,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 2.4884722232818604,
+      "learning_rate": 1.1960916442048519e-05,
+      "loss": 0.0858,
+      "step": 2770
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 2.3672924041748047,
+      "learning_rate": 1.1623989218328842e-05,
+      "loss": 0.0832,
+      "step": 2780
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 1.5933914184570312,
+      "learning_rate": 1.1287061994609164e-05,
+      "loss": 0.0825,
+      "step": 2790
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 1.7233848571777344,
+      "learning_rate": 1.0950134770889488e-05,
+      "loss": 0.0861,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 0.8971027135848999,
+      "learning_rate": 1.0613207547169812e-05,
+      "loss": 0.0878,
+      "step": 2810
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 1.5226572751998901,
+      "learning_rate": 1.0276280323450135e-05,
+      "loss": 0.0774,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 1.4949426651000977,
+      "learning_rate": 9.939353099730459e-06,
+      "loss": 0.0782,
+      "step": 2830
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 1.4113121032714844,
+      "learning_rate": 9.602425876010781e-06,
+      "loss": 0.0896,
+      "step": 2840
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 1.9517556428909302,
+      "learning_rate": 9.265498652291107e-06,
+      "loss": 0.0838,
+      "step": 2850
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 1.0090752840042114,
+      "learning_rate": 8.92857142857143e-06,
+      "loss": 0.0768,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 1.0155930519104004,
+      "learning_rate": 8.591644204851752e-06,
+      "loss": 0.0771,
+      "step": 2870
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 1.2357333898544312,
+      "learning_rate": 8.254716981132076e-06,
+      "loss": 0.088,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 1.2568587064743042,
+      "learning_rate": 7.9177897574124e-06,
+      "loss": 0.0806,
+      "step": 2890
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 0.9047762751579285,
+      "learning_rate": 7.580862533692723e-06,
+      "loss": 0.0883,
+      "step": 2900
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 1.890642523765564,
+      "learning_rate": 7.243935309973046e-06,
+      "loss": 0.0865,
+      "step": 2910
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 2.2417125701904297,
+      "learning_rate": 6.90700808625337e-06,
+      "loss": 0.086,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 1.3787548542022705,
+      "learning_rate": 6.570080862533692e-06,
+      "loss": 0.0825,
+      "step": 2930
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 0.8920753598213196,
+      "learning_rate": 6.233153638814016e-06,
+      "loss": 0.0821,
+      "step": 2940
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 1.1256639957427979,
+      "learning_rate": 5.89622641509434e-06,
+      "loss": 0.0861,
+      "step": 2950
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 1.4777594804763794,
+      "learning_rate": 5.5592991913746634e-06,
+      "loss": 0.0845,
+      "step": 2960
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 1.1454410552978516,
+      "learning_rate": 5.222371967654987e-06,
+      "loss": 0.0936,
+      "step": 2970
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 1.2177817821502686,
+      "learning_rate": 4.88544474393531e-06,
+      "loss": 0.0826,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 1.397789716720581,
+      "learning_rate": 4.548517520215634e-06,
+      "loss": 0.0839,
+      "step": 2990
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 1.3551985025405884,
+      "learning_rate": 4.211590296495957e-06,
+      "loss": 0.0865,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 1.4842958450317383,
+      "learning_rate": 3.8746630727762805e-06,
+      "loss": 0.0784,
+      "step": 3010
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 1.2862508296966553,
+      "learning_rate": 3.5377358490566038e-06,
+      "loss": 0.0857,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 1.299244999885559,
+      "learning_rate": 3.200808625336928e-06,
+      "loss": 0.0851,
+      "step": 3030
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 1.7346903085708618,
+      "learning_rate": 2.8638814016172507e-06,
+      "loss": 0.0863,
+      "step": 3040
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 1.687151312828064,
+      "learning_rate": 2.5269541778975744e-06,
+      "loss": 0.0858,
+      "step": 3050
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 2.20270037651062,
+      "learning_rate": 2.1900269541778976e-06,
+      "loss": 0.088,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 0.9984031319618225,
+      "learning_rate": 1.853099730458221e-06,
+      "loss": 0.0804,
+      "step": 3070
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 0.9710437655448914,
+      "learning_rate": 1.5161725067385445e-06,
+      "loss": 0.0829,
+      "step": 3080
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 1.1395683288574219,
+      "learning_rate": 1.179245283018868e-06,
+      "loss": 0.0841,
+      "step": 3090
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 1.1039100885391235,
+      "learning_rate": 8.423180592991913e-07,
+      "loss": 0.0763,
+      "step": 3100
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 1.502783179283142,
+      "learning_rate": 5.053908355795148e-07,
+      "loss": 0.0907,
+      "step": 3110
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 1.4095673561096191,
+      "learning_rate": 1.684636118598383e-07,
+      "loss": 0.0772,
+      "step": 3120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 3125,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3125/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:854e9cec59f8e6c96cd43644cdac1ccee3b4bf13a4d82830503cdf1396dbeabc
+size 5304

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "gpt2-medium",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "n_special": 0,
+  "predict_special_tokens": true,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "transformers_version": "4.45.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_logs.csv

@@ -0,0 +1,314 @@
+step,training_loss,grad_norm
+10,22.4391,5.000000476837158
+20,18.5406,5.000000476837158
+30,18.8756,5.0
+40,18.0104,4.999999046325684
+50,15.1106,4.999999523162842
+60,14.5177,4.999999523162842
+70,12.4196,5.0
+80,10.4176,5.0
+90,9.3694,4.999999523162842
+100,8.6032,4.999999046325684
+110,6.5879,4.999999046325684
+120,6.7468,4.999999523162842
+130,5.6174,4.999999523162842
+140,5.1907,5.0
+150,4.9405,4.999999523162842
+160,4.4357,4.999999046325684
+170,4.1208,5.0
+180,4.103,4.999999046325684
+190,4.0038,4.999999046325684
+200,3.4375,5.0
+210,3.4134,4.999999046325684
+220,3.1147,5.0
+230,3.2726,4.999999523162842
+240,3.0596,4.999999523162842
+250,2.9703,4.999999523162842
+260,2.9099,4.999999523162842
+270,3.084,4.999999523162842
+280,2.7716,4.999999523162842
+290,2.5286,4.999999046325684
+300,2.9801,4.999999046325684
+310,2.6343,4.999999046325684
+320,2.5548,4.999999046325684
+330,2.3042,4.999999523162842
+340,2.1663,4.999999523162842
+350,2.2429,5.0
+360,1.9972,4.999999523162842
+370,1.9431,4.999999523162842
+380,1.9534,5.0
+390,1.7552,4.999999523162842
+400,1.7837,5.0
+410,1.6729,5.0
+420,1.5536,4.999998569488525
+430,1.6502,4.999999046325684
+440,1.4372,5.0
+450,1.43,4.999999046325684
+460,1.4757,4.999999046325684
+470,1.323,4.999999046325684
+480,1.257,4.999998569488525
+490,1.2685,4.999999046325684
+500,1.3465,4.999999523162842
+510,1.1891,4.999999523162842
+520,1.271,4.999999046325684
+530,1.1265,4.999999046325684
+540,1.2308,4.999999523162842
+550,1.1909,4.999999523162842
+560,1.1423,4.743152618408203
+570,1.1738,4.999999523162842
+580,1.1288,4.999999523162842
+590,1.0923,4.999999523162842
+600,1.0964,4.999999046325684
+610,0.9797,4.999999523162842
+620,1.0295,4.760279655456543
+630,0.9473,4.999999046325684
+640,1.0278,4.999999523162842
+650,0.9588,5.0
+660,0.954,4.999999046325684
+670,0.9665,4.999999046325684
+680,0.9109,4.999999046325684
+690,0.9,4.999999046325684
+700,0.8534,4.999999523162842
+710,0.8176,4.999999046325684
+720,0.8267,4.999999046325684
+730,0.8188,4.478681564331055
+740,0.8477,4.999999523162842
+750,0.789,4.999999046325684
+760,0.7381,4.999999046325684
+770,0.754,4.999999523162842
+780,0.7388,4.999999523162842
+790,0.8036,5.0
+800,0.803,4.999999523162842
+810,0.7286,4.999999046325684
+820,0.6898,4.999999523162842
+830,0.6662,4.999999046325684
+840,0.6753,4.999999523162842
+850,0.6469,4.999999523162842
+860,0.6496,4.999999523162842
+870,0.6454,4.999999046325684
+880,0.5762,4.928877830505371
+890,0.6006,4.999999046325684
+900,0.6075,4.999999523162842
+910,0.5837,4.305392742156982
+920,0.5447,4.999999046325684
+930,0.5516,5.0
+940,0.5024,4.88629150390625
+950,0.571,4.116428375244141
+960,0.4532,4.824604034423828
+970,0.4883,3.117197275161743
+980,0.4787,4.999999046325684
+990,0.4637,4.999999523162842
+1000,0.4781,4.999999046325684
+1010,0.4286,4.999999046325684
+1020,0.4412,4.999999046325684
+1030,0.4406,3.544337749481201
+1040,0.4078,4.349572658538818
+1050,0.4726,4.470551013946533
+1060,0.4033,4.85098934173584
+1070,0.3966,4.999999046325684
+1080,0.4374,3.946486711502075
+1090,0.3775,3.7488667964935303
+1100,0.3846,3.612640857696533
+1110,0.3444,4.999999046325684
+1120,0.3635,4.179413795471191
+1130,0.3313,4.016063213348389
+1140,0.3312,3.057584762573242
+1150,0.3141,2.5960428714752197
+1160,0.3122,3.4362008571624756
+1170,0.3085,4.949530601501465
+1180,0.3028,3.274339199066162
+1190,0.3128,4.999999046325684
+1200,0.253,3.0627970695495605
+1210,0.2982,3.8672235012054443
+1220,0.2763,3.373741388320923
+1230,0.2833,3.5057952404022217
+1240,0.284,3.149587631225586
+1250,0.2553,3.716684103012085
+1260,0.2414,2.4263858795166016
+1270,0.2542,3.104553699493408
+1280,0.2442,2.915567636489868
+1290,0.2833,3.980184316635132
+1300,0.248,4.143345355987549
+1310,0.2426,1.8089429140090942
+1320,0.246,1.8527010679244995
+1330,0.2405,3.4966213703155518
+1340,0.2713,2.331099510192871
+1350,0.2202,3.4839956760406494
+1360,0.246,2.3249175548553467
+1370,0.2126,3.72847318649292
+1380,0.2059,2.6438794136047363
+1390,0.2061,3.9793527126312256
+1400,0.2063,3.324707508087158
+1410,0.2234,3.0844309329986572
+1420,0.2032,3.1997532844543457
+1430,0.2186,2.557157278060913
+1440,0.1934,2.0848405361175537
+1450,0.191,1.8372052907943726
+1460,0.1689,2.1945323944091797
+1470,0.1901,3.432448625564575
+1480,0.2023,2.2553012371063232
+1490,0.1777,2.1100335121154785
+1500,0.1672,1.9357300996780396
+1510,0.1865,2.9728143215179443
+1520,0.1942,3.1837971210479736
+1530,0.189,3.62715220451355
+1540,0.1781,2.986555576324463
+1550,0.1618,1.4352009296417236
+1560,0.1739,2.6908838748931885
+1570,0.1782,2.1369287967681885
+1580,0.1669,1.9476417303085327
+1590,0.1634,2.5264928340911865
+1600,0.1763,2.2296202182769775
+1610,0.1863,1.9065362215042114
+1620,0.1745,2.1693010330200195
+1630,0.1465,1.8745787143707275
+1640,0.1525,2.0987377166748047
+1650,0.1525,2.405745506286621
+1660,0.1447,3.3560268878936768
+1670,0.1689,1.3180267810821533
+1680,0.1489,1.4368494749069214
+1690,0.1368,1.5566891431808472
+1700,0.1445,2.221500873565674
+1710,0.1426,2.9313652515411377
+1720,0.138,1.836387038230896
+1730,0.1449,1.5651477575302124
+1740,0.1438,3.2158570289611816
+1750,0.1423,1.5862022638320923
+1760,0.1469,1.6251810789108276
+1770,0.13,1.594459056854248
+1780,0.1309,1.554364562034607
+1790,0.1504,2.1495261192321777
+1800,0.1499,2.061843156814575
+1810,0.1393,1.517041802406311
+1820,0.1278,1.455052137374878
+1830,0.141,2.3193039894104004
+1840,0.1361,1.6179640293121338
+1850,0.1333,2.4831204414367676
+1860,0.1195,1.112313985824585
+1870,0.1232,1.4801064729690552
+1880,0.1389,2.4858882427215576
+1890,0.1236,1.9607441425323486
+1900,0.1297,2.4481492042541504
+1910,0.134,1.8709760904312134
+1920,0.1259,3.084176778793335
+1930,0.1157,1.546714186668396
+1940,0.1185,2.207704782485962
+1950,0.1222,1.6168131828308105
+1960,0.1156,3.353578567504883
+1970,0.1218,2.5373737812042236
+1980,0.1362,4.4312968254089355
+1990,0.1139,1.7041658163070679
+2000,0.1314,2.0976693630218506
+2010,0.1351,1.9576454162597656
+2020,0.1123,1.4846729040145874
+2030,0.1168,1.9108271598815918
+2040,0.1225,1.186391830444336
+2050,0.1226,2.4576640129089355
+2060,0.1242,1.863196849822998
+2070,0.1098,1.3311208486557007
+2080,0.1199,1.780318021774292
+2090,0.1051,2.5270535945892334
+2100,0.1034,1.394476294517517
+2110,0.1265,1.3372985124588013
+2120,0.1038,2.175616979598999
+2130,0.1243,2.5146851539611816
+2140,0.1136,1.9544363021850586
+2150,0.1087,1.6316698789596558
+2160,0.103,1.7699568271636963
+2170,0.1147,1.3398338556289673
+2180,0.1093,1.3555325269699097
+2190,0.1083,1.2930587530136108
+2200,0.0958,1.017906665802002
+2210,0.1284,2.231780529022217
+2220,0.1039,1.7175219058990479
+2230,0.1059,1.588964581489563
+2240,0.1012,1.4871183633804321
+2250,0.0946,1.933687686920166
+2260,0.096,1.170185923576355
+2270,0.1001,1.33351469039917
+2280,0.0855,1.051422119140625
+2290,0.0903,2.476947069168091
+2300,0.101,1.5270748138427734
+2310,0.0901,1.573987364768982
+2320,0.1053,1.211846947669983
+2330,0.0956,1.0927597284317017
+2340,0.1084,1.5684590339660645
+2350,0.0986,1.7778979539871216
+2360,0.0982,1.2901784181594849
+2370,0.1115,1.230678915977478
+2380,0.0967,1.37783944606781
+2390,0.0984,1.5255314111709595
+2400,0.0987,2.0732505321502686
+2410,0.0944,1.2267639636993408
+2420,0.0923,2.0225038528442383
+2430,0.096,2.311718463897705
+2440,0.0935,2.0240392684936523
+2450,0.0886,1.575096607208252
+2460,0.0971,1.8266900777816772
+2470,0.0897,1.2168657779693604
+2480,0.0942,1.8267066478729248
+2490,0.0921,1.4711161851882935
+2500,0.0875,1.9106966257095337
+2510,0.0929,2.4690868854522705
+2520,0.0876,1.4428863525390625
+2530,0.0915,1.3941131830215454
+2540,0.09,1.9686850309371948
+2550,0.0939,1.6172926425933838
+2560,0.0876,1.93358314037323
+2570,0.1017,1.436843752861023
+2580,0.0884,1.1019058227539062
+2590,0.0906,1.6116135120391846
+2600,0.0975,1.0761088132858276
+2610,0.0856,1.3172565698623657
+2620,0.0859,1.0831719636917114
+2630,0.0865,2.260585308074951
+2640,0.0959,1.5628427267074585
+2650,0.0862,1.4175759553909302
+2660,0.0882,1.6256355047225952
+2670,0.0939,1.1748156547546387
+2680,0.0822,1.1186449527740479
+2690,0.0853,1.2344774007797241
+2700,0.0934,1.4567972421646118
+2710,0.0901,1.1431821584701538
+2720,0.0849,1.644754409790039
+2730,0.0846,0.9929800033569336
+2740,0.0824,1.2569183111190796
+2750,0.0912,1.3616032600402832
+2760,0.1324,1.4828603267669678
+2770,0.0858,2.4884722232818604
+2780,0.0832,2.3672924041748047
+2790,0.0825,1.5933914184570312
+2800,0.0861,1.7233848571777344
+2810,0.0878,0.8971027135848999
+2820,0.0774,1.5226572751998901
+2830,0.0782,1.4949426651000977
+2840,0.0896,1.4113121032714844
+2850,0.0838,1.9517556428909302
+2860,0.0768,1.0090752840042114
+2870,0.0771,1.0155930519104004
+2880,0.088,1.2357333898544312
+2890,0.0806,1.2568587064743042
+2900,0.0883,0.9047762751579285
+2910,0.0865,1.890642523765564
+2920,0.086,2.2417125701904297
+2930,0.0825,1.3787548542022705
+2940,0.0821,0.8920753598213196
+2950,0.0861,1.1256639957427979
+2960,0.0845,1.4777594804763794
+2970,0.0936,1.1454410552978516
+2980,0.0826,1.2177817821502686
+2990,0.0839,1.397789716720581
+3000,0.0865,1.3551985025405884
+3010,0.0784,1.4842958450317383
+3020,0.0857,1.2862508296966553
+3030,0.0851,1.299244999885559
+3040,0.0863,1.7346903085708618
+3050,0.0858,1.687151312828064
+3060,0.088,2.20270037651062
+3070,0.0804,0.9984031319618225
+3080,0.0829,0.9710437655448914
+3090,0.0841,1.1395683288574219
+3100,0.0763,1.1039100885391235
+3110,0.0907,1.502783179283142
+3120,0.0772,1.4095673561096191
+3125,nan,nan