Upload folder using huggingface_hub

README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cea55d0009077da4c17c079795c8d6965e0a2a187faa63010acf628ad2d80407
+size 25172088

checkpoint-3125/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

checkpoint-3125/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-3125/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cea55d0009077da4c17c079795c8d6965e0a2a187faa63010acf628ad2d80407
+size 25172088

checkpoint-3125/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df3ff59a567c307b63a9e3d4cefc42c823c19a6e108e599c9b352eb0411ee158
+size 50372538

checkpoint-3125/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5994634e90ce1b45d012a45568063be05fea876e791cd66b48a4efc924164b2
+size 14244

checkpoint-3125/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e42e57efa8b95889e87232fe6ea0805483ed6c8a2d35bb60230715b624ff5bda
+size 1064

checkpoint-3125/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-3125/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-3125/trainer_state.json

@@ -0,0 +1,2217 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 3125,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 10.000000953674316,
+      "learning_rate": 6.369426751592357e-06,
+      "loss": 22.4391,
+      "step": 10
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 1.2738853503184714e-05,
+      "loss": 18.5398,
+      "step": 20
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 10.0,
+      "learning_rate": 1.910828025477707e-05,
+      "loss": 18.8696,
+      "step": 30
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 2.5477707006369428e-05,
+      "loss": 17.9832,
+      "step": 40
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 10.0,
+      "learning_rate": 3.184713375796178e-05,
+      "loss": 15.0181,
+      "step": 50
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 3.821656050955414e-05,
+      "loss": 14.2746,
+      "step": 60
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 10.0,
+      "learning_rate": 4.45859872611465e-05,
+      "loss": 11.9335,
+      "step": 70
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 10.0,
+      "learning_rate": 5.0955414012738855e-05,
+      "loss": 9.6611,
+      "step": 80
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 10.0,
+      "learning_rate": 5.732484076433121e-05,
+      "loss": 8.2423,
+      "step": 90
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 6.369426751592356e-05,
+      "loss": 7.0975,
+      "step": 100
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 10.0,
+      "learning_rate": 7.006369426751592e-05,
+      "loss": 5.2427,
+      "step": 110
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 7.643312101910829e-05,
+      "loss": 5.0576,
+      "step": 120
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 8.280254777070065e-05,
+      "loss": 4.3215,
+      "step": 130
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 8.9171974522293e-05,
+      "loss": 3.7479,
+      "step": 140
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 9.422039985656738,
+      "learning_rate": 9.554140127388536e-05,
+      "loss": 3.5213,
+      "step": 150
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 9.989892183288411e-05,
+      "loss": 3.2425,
+      "step": 160
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 9.351380348205566,
+      "learning_rate": 9.956199460916442e-05,
+      "loss": 2.8891,
+      "step": 170
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 8.307005882263184,
+      "learning_rate": 9.922506738544474e-05,
+      "loss": 2.783,
+      "step": 180
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 8.963129997253418,
+      "learning_rate": 9.888814016172507e-05,
+      "loss": 2.7527,
+      "step": 190
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 8.570749282836914,
+      "learning_rate": 9.85512129380054e-05,
+      "loss": 2.4355,
+      "step": 200
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 8.063360214233398,
+      "learning_rate": 9.821428571428572e-05,
+      "loss": 2.4091,
+      "step": 210
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 8.824004173278809,
+      "learning_rate": 9.787735849056603e-05,
+      "loss": 2.1608,
+      "step": 220
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 7.373661994934082,
+      "learning_rate": 9.754043126684636e-05,
+      "loss": 2.2534,
+      "step": 230
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 8.307866096496582,
+      "learning_rate": 9.720350404312669e-05,
+      "loss": 2.1537,
+      "step": 240
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 5.856404781341553,
+      "learning_rate": 9.686657681940702e-05,
+      "loss": 1.9916,
+      "step": 250
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 8.64548397064209,
+      "learning_rate": 9.652964959568734e-05,
+      "loss": 1.9904,
+      "step": 260
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 7.271899223327637,
+      "learning_rate": 9.619272237196765e-05,
+      "loss": 2.0503,
+      "step": 270
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 6.368246555328369,
+      "learning_rate": 9.585579514824798e-05,
+      "loss": 1.9192,
+      "step": 280
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 6.038721561431885,
+      "learning_rate": 9.551886792452831e-05,
+      "loss": 1.6597,
+      "step": 290
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 6.260958194732666,
+      "learning_rate": 9.518194070080863e-05,
+      "loss": 1.9203,
+      "step": 300
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 7.62371301651001,
+      "learning_rate": 9.484501347708896e-05,
+      "loss": 1.8547,
+      "step": 310
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 7.491437911987305,
+      "learning_rate": 9.450808625336927e-05,
+      "loss": 1.7249,
+      "step": 320
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 7.197510719299316,
+      "learning_rate": 9.41711590296496e-05,
+      "loss": 1.5601,
+      "step": 330
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 6.928409099578857,
+      "learning_rate": 9.383423180592993e-05,
+      "loss": 1.6026,
+      "step": 340
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 6.700569152832031,
+      "learning_rate": 9.349730458221025e-05,
+      "loss": 1.5981,
+      "step": 350
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 6.20810604095459,
+      "learning_rate": 9.316037735849057e-05,
+      "loss": 1.4863,
+      "step": 360
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 6.265394687652588,
+      "learning_rate": 9.282345013477089e-05,
+      "loss": 1.5054,
+      "step": 370
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 7.249783039093018,
+      "learning_rate": 9.248652291105122e-05,
+      "loss": 1.4892,
+      "step": 380
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 6.698989391326904,
+      "learning_rate": 9.214959568733154e-05,
+      "loss": 1.3346,
+      "step": 390
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 6.408270359039307,
+      "learning_rate": 9.181266846361186e-05,
+      "loss": 1.269,
+      "step": 400
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 6.221065521240234,
+      "learning_rate": 9.14757412398922e-05,
+      "loss": 1.2479,
+      "step": 410
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 6.697783470153809,
+      "learning_rate": 9.113881401617251e-05,
+      "loss": 1.1823,
+      "step": 420
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 6.298666000366211,
+      "learning_rate": 9.080188679245284e-05,
+      "loss": 1.2215,
+      "step": 430
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 5.8949875831604,
+      "learning_rate": 9.046495956873315e-05,
+      "loss": 1.2094,
+      "step": 440
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 4.865890026092529,
+      "learning_rate": 9.012803234501348e-05,
+      "loss": 1.1668,
+      "step": 450
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 4.905447006225586,
+      "learning_rate": 8.979110512129381e-05,
+      "loss": 1.0941,
+      "step": 460
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 5.108652591705322,
+      "learning_rate": 8.945417789757413e-05,
+      "loss": 1.081,
+      "step": 470
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 4.898155212402344,
+      "learning_rate": 8.911725067385444e-05,
+      "loss": 1.0708,
+      "step": 480
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 4.2186455726623535,
+      "learning_rate": 8.878032345013477e-05,
+      "loss": 0.9574,
+      "step": 490
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 4.5638957023620605,
+      "learning_rate": 8.84433962264151e-05,
+      "loss": 1.1016,
+      "step": 500
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 5.078132152557373,
+      "learning_rate": 8.810646900269543e-05,
+      "loss": 0.9938,
+      "step": 510
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 6.263752460479736,
+      "learning_rate": 8.776954177897575e-05,
+      "loss": 1.0042,
+      "step": 520
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 4.613877773284912,
+      "learning_rate": 8.743261455525606e-05,
+      "loss": 0.8717,
+      "step": 530
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 5.016404151916504,
+      "learning_rate": 8.709568733153639e-05,
+      "loss": 0.9252,
+      "step": 540
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 4.927691459655762,
+      "learning_rate": 8.675876010781672e-05,
+      "loss": 0.9297,
+      "step": 550
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 4.506190299987793,
+      "learning_rate": 8.642183288409704e-05,
+      "loss": 0.8883,
+      "step": 560
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 5.3836350440979,
+      "learning_rate": 8.608490566037735e-05,
+      "loss": 0.8302,
+      "step": 570
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 4.673761367797852,
+      "learning_rate": 8.574797843665768e-05,
+      "loss": 0.8401,
+      "step": 580
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 3.7030556201934814,
+      "learning_rate": 8.541105121293801e-05,
+      "loss": 0.888,
+      "step": 590
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 4.952301025390625,
+      "learning_rate": 8.507412398921834e-05,
+      "loss": 0.8483,
+      "step": 600
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 4.4545769691467285,
+      "learning_rate": 8.473719676549866e-05,
+      "loss": 0.7925,
+      "step": 610
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 4.0674896240234375,
+      "learning_rate": 8.440026954177897e-05,
+      "loss": 0.7347,
+      "step": 620
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 6.8568034172058105,
+      "learning_rate": 8.40633423180593e-05,
+      "loss": 0.7413,
+      "step": 630
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 6.715765476226807,
+      "learning_rate": 8.372641509433963e-05,
+      "loss": 0.7714,
+      "step": 640
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 4.419723033905029,
+      "learning_rate": 8.338948787061996e-05,
+      "loss": 0.7337,
+      "step": 650
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 4.291261672973633,
+      "learning_rate": 8.305256064690027e-05,
+      "loss": 0.6854,
+      "step": 660
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 5.974247932434082,
+      "learning_rate": 8.271563342318059e-05,
+      "loss": 0.7224,
+      "step": 670
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 4.334232807159424,
+      "learning_rate": 8.237870619946092e-05,
+      "loss": 0.6737,
+      "step": 680
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 4.699735164642334,
+      "learning_rate": 8.204177897574125e-05,
+      "loss": 0.6505,
+      "step": 690
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 4.12949275970459,
+      "learning_rate": 8.170485175202158e-05,
+      "loss": 0.6696,
+      "step": 700
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 3.9896364212036133,
+      "learning_rate": 8.136792452830189e-05,
+      "loss": 0.6587,
+      "step": 710
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 4.438923358917236,
+      "learning_rate": 8.103099730458221e-05,
+      "loss": 0.6189,
+      "step": 720
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 3.4031989574432373,
+      "learning_rate": 8.069407008086254e-05,
+      "loss": 0.6249,
+      "step": 730
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 5.202292442321777,
+      "learning_rate": 8.035714285714287e-05,
+      "loss": 0.6462,
+      "step": 740
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 5.27857780456543,
+      "learning_rate": 8.002021563342318e-05,
+      "loss": 0.5842,
+      "step": 750
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 4.678514003753662,
+      "learning_rate": 7.968328840970351e-05,
+      "loss": 0.5659,
+      "step": 760
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 4.691657543182373,
+      "learning_rate": 7.934636118598383e-05,
+      "loss": 0.5833,
+      "step": 770
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 7.131917476654053,
+      "learning_rate": 7.900943396226416e-05,
+      "loss": 0.5958,
+      "step": 780
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 5.070577144622803,
+      "learning_rate": 7.867250673854449e-05,
+      "loss": 0.6307,
+      "step": 790
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 4.827574253082275,
+      "learning_rate": 7.83355795148248e-05,
+      "loss": 0.6064,
+      "step": 800
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 3.8818304538726807,
+      "learning_rate": 7.799865229110512e-05,
+      "loss": 0.5218,
+      "step": 810
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 6.581722736358643,
+      "learning_rate": 7.766172506738545e-05,
+      "loss": 0.5342,
+      "step": 820
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 9.561187744140625,
+      "learning_rate": 7.732479784366577e-05,
+      "loss": 0.5167,
+      "step": 830
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 6.698048114776611,
+      "learning_rate": 7.69878706199461e-05,
+      "loss": 0.5674,
+      "step": 840
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 5.142230987548828,
+      "learning_rate": 7.665094339622642e-05,
+      "loss": 0.5432,
+      "step": 850
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 5.226676940917969,
+      "learning_rate": 7.631401617250674e-05,
+      "loss": 0.557,
+      "step": 860
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 5.462721824645996,
+      "learning_rate": 7.597708894878706e-05,
+      "loss": 0.4969,
+      "step": 870
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 4.489198207855225,
+      "learning_rate": 7.56401617250674e-05,
+      "loss": 0.5004,
+      "step": 880
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 4.706643104553223,
+      "learning_rate": 7.530323450134771e-05,
+      "loss": 0.4465,
+      "step": 890
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 7.614391803741455,
+      "learning_rate": 7.496630727762804e-05,
+      "loss": 0.4989,
+      "step": 900
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 3.3230834007263184,
+      "learning_rate": 7.462938005390835e-05,
+      "loss": 0.4978,
+      "step": 910
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 4.60692834854126,
+      "learning_rate": 7.429245283018868e-05,
+      "loss": 0.4477,
+      "step": 920
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 4.93078088760376,
+      "learning_rate": 7.395552560646901e-05,
+      "loss": 0.4573,
+      "step": 930
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 3.973932981491089,
+      "learning_rate": 7.361859838274933e-05,
+      "loss": 0.4262,
+      "step": 940
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 3.4613964557647705,
+      "learning_rate": 7.328167115902966e-05,
+      "loss": 0.4804,
+      "step": 950
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 3.4361777305603027,
+      "learning_rate": 7.294474393530997e-05,
+      "loss": 0.4037,
+      "step": 960
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 3.738327980041504,
+      "learning_rate": 7.26078167115903e-05,
+      "loss": 0.4436,
+      "step": 970
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 5.539529323577881,
+      "learning_rate": 7.227088948787062e-05,
+      "loss": 0.4489,
+      "step": 980
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 3.828397512435913,
+      "learning_rate": 7.193396226415095e-05,
+      "loss": 0.4346,
+      "step": 990
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 4.032661437988281,
+      "learning_rate": 7.159703504043128e-05,
+      "loss": 0.4615,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 3.523750066757202,
+      "learning_rate": 7.126010781671159e-05,
+      "loss": 0.4137,
+      "step": 1010
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 5.352044582366943,
+      "learning_rate": 7.092318059299192e-05,
+      "loss": 0.4397,
+      "step": 1020
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 3.491257667541504,
+      "learning_rate": 7.058625336927224e-05,
+      "loss": 0.4411,
+      "step": 1030
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 2.7892098426818848,
+      "learning_rate": 7.024932614555257e-05,
+      "loss": 0.3997,
+      "step": 1040
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 3.231407880783081,
+      "learning_rate": 6.99123989218329e-05,
+      "loss": 0.4118,
+      "step": 1050
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 4.61359977722168,
+      "learning_rate": 6.957547169811321e-05,
+      "loss": 0.3631,
+      "step": 1060
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 7.804078578948975,
+      "learning_rate": 6.923854447439353e-05,
+      "loss": 0.399,
+      "step": 1070
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 5.450227737426758,
+      "learning_rate": 6.890161725067386e-05,
+      "loss": 0.4628,
+      "step": 1080
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 4.55374002456665,
+      "learning_rate": 6.856469002695418e-05,
+      "loss": 0.4149,
+      "step": 1090
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 2.5150911808013916,
+      "learning_rate": 6.822776280323451e-05,
+      "loss": 0.3892,
+      "step": 1100
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 3.4889912605285645,
+      "learning_rate": 6.789083557951483e-05,
+      "loss": 0.3874,
+      "step": 1110
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 4.045688629150391,
+      "learning_rate": 6.755390835579514e-05,
+      "loss": 0.4126,
+      "step": 1120
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 5.061463356018066,
+      "learning_rate": 6.721698113207547e-05,
+      "loss": 0.3564,
+      "step": 1130
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 3.681854248046875,
+      "learning_rate": 6.68800539083558e-05,
+      "loss": 0.4036,
+      "step": 1140
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 4.347032070159912,
+      "learning_rate": 6.654312668463612e-05,
+      "loss": 0.4049,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 7.9803314208984375,
+      "learning_rate": 6.620619946091643e-05,
+      "loss": 0.3708,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 5.374613285064697,
+      "learning_rate": 6.586927223719676e-05,
+      "loss": 0.376,
+      "step": 1170
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 3.0691614151000977,
+      "learning_rate": 6.553234501347709e-05,
+      "loss": 0.371,
+      "step": 1180
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 4.836121082305908,
+      "learning_rate": 6.519541778975742e-05,
+      "loss": 0.3681,
+      "step": 1190
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 3.778252124786377,
+      "learning_rate": 6.485849056603774e-05,
+      "loss": 0.3379,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 3.3291821479797363,
+      "learning_rate": 6.452156334231805e-05,
+      "loss": 0.3627,
+      "step": 1210
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 3.522620916366577,
+      "learning_rate": 6.418463611859838e-05,
+      "loss": 0.3423,
+      "step": 1220
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 3.283186912536621,
+      "learning_rate": 6.384770889487871e-05,
+      "loss": 0.3301,
+      "step": 1230
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 4.433807849884033,
+      "learning_rate": 6.351078167115904e-05,
+      "loss": 0.3468,
+      "step": 1240
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 3.97624135017395,
+      "learning_rate": 6.317385444743936e-05,
+      "loss": 0.3376,
+      "step": 1250
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 3.079582929611206,
+      "learning_rate": 6.283692722371967e-05,
+      "loss": 0.3032,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 4.162789821624756,
+      "learning_rate": 6.25e-05,
+      "loss": 0.3452,
+      "step": 1270
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 4.1800665855407715,
+      "learning_rate": 6.216307277628033e-05,
+      "loss": 0.3351,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 4.206212520599365,
+      "learning_rate": 6.182614555256066e-05,
+      "loss": 0.3579,
+      "step": 1290
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 3.900646924972534,
+      "learning_rate": 6.148921832884098e-05,
+      "loss": 0.3328,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 5.285961627960205,
+      "learning_rate": 6.115229110512129e-05,
+      "loss": 0.345,
+      "step": 1310
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 2.616771697998047,
+      "learning_rate": 6.081536388140162e-05,
+      "loss": 0.321,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 3.762444496154785,
+      "learning_rate": 6.047843665768195e-05,
+      "loss": 0.3188,
+      "step": 1330
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 3.244722366333008,
+      "learning_rate": 6.0141509433962265e-05,
+      "loss": 0.3299,
+      "step": 1340
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 3.7463319301605225,
+      "learning_rate": 5.980458221024259e-05,
+      "loss": 0.3063,
+      "step": 1350
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 2.2992026805877686,
+      "learning_rate": 5.9467654986522916e-05,
+      "loss": 0.3218,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 3.3672423362731934,
+      "learning_rate": 5.913072776280324e-05,
+      "loss": 0.3295,
+      "step": 1370
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 3.5933585166931152,
+      "learning_rate": 5.879380053908357e-05,
+      "loss": 0.282,
+      "step": 1380
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 5.269179821014404,
+      "learning_rate": 5.8456873315363884e-05,
+      "loss": 0.3113,
+      "step": 1390
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 2.99794340133667,
+      "learning_rate": 5.8119946091644206e-05,
+      "loss": 0.2977,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 3.103388547897339,
+      "learning_rate": 5.7783018867924535e-05,
+      "loss": 0.2954,
+      "step": 1410
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 2.7087464332580566,
+      "learning_rate": 5.744609164420486e-05,
+      "loss": 0.2779,
+      "step": 1420
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 4.316291809082031,
+      "learning_rate": 5.710916442048517e-05,
+      "loss": 0.3277,
+      "step": 1430
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 3.2305455207824707,
+      "learning_rate": 5.6772237196765496e-05,
+      "loss": 0.2738,
+      "step": 1440
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 2.637054443359375,
+      "learning_rate": 5.6435309973045825e-05,
+      "loss": 0.2764,
+      "step": 1450
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 2.727341890335083,
+      "learning_rate": 5.609838274932615e-05,
+      "loss": 0.2624,
+      "step": 1460
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 3.334686756134033,
+      "learning_rate": 5.5761455525606476e-05,
+      "loss": 0.2889,
+      "step": 1470
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 2.98366641998291,
+      "learning_rate": 5.542452830188679e-05,
+      "loss": 0.3006,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 3.3115627765655518,
+      "learning_rate": 5.5087601078167114e-05,
+      "loss": 0.2628,
+      "step": 1490
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 4.2003583908081055,
+      "learning_rate": 5.4750673854447444e-05,
+      "loss": 0.2607,
+      "step": 1500
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 6.368921756744385,
+      "learning_rate": 5.4413746630727766e-05,
+      "loss": 0.2869,
+      "step": 1510
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 6.977168083190918,
+      "learning_rate": 5.407681940700808e-05,
+      "loss": 0.2757,
+      "step": 1520
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 2.90818452835083,
+      "learning_rate": 5.373989218328841e-05,
+      "loss": 0.2545,
+      "step": 1530
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 3.735295534133911,
+      "learning_rate": 5.340296495956873e-05,
+      "loss": 0.2632,
+      "step": 1540
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 3.361546039581299,
+      "learning_rate": 5.306603773584906e-05,
+      "loss": 0.2347,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 4.228140830993652,
+      "learning_rate": 5.2729110512129385e-05,
+      "loss": 0.2601,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 4.071884632110596,
+      "learning_rate": 5.23921832884097e-05,
+      "loss": 0.2604,
+      "step": 1570
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 2.7090647220611572,
+      "learning_rate": 5.205525606469003e-05,
+      "loss": 0.2353,
+      "step": 1580
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 5.145593166351318,
+      "learning_rate": 5.171832884097035e-05,
+      "loss": 0.2311,
+      "step": 1590
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 3.904247522354126,
+      "learning_rate": 5.138140161725068e-05,
+      "loss": 0.2466,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 1.8991984128952026,
+      "learning_rate": 5.1044474393531e-05,
+      "loss": 0.267,
+      "step": 1610
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 3.3171005249023438,
+      "learning_rate": 5.070754716981132e-05,
+      "loss": 0.2648,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 2.8795769214630127,
+      "learning_rate": 5.037061994609165e-05,
+      "loss": 0.2293,
+      "step": 1630
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 3.4193594455718994,
+      "learning_rate": 5.003369272237197e-05,
+      "loss": 0.2243,
+      "step": 1640
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 3.470912218093872,
+      "learning_rate": 4.969676549865229e-05,
+      "loss": 0.2339,
+      "step": 1650
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 4.2241926193237305,
+      "learning_rate": 4.9359838274932616e-05,
+      "loss": 0.2346,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 2.4578678607940674,
+      "learning_rate": 4.902291105121294e-05,
+      "loss": 0.2162,
+      "step": 1670
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 2.8263041973114014,
+      "learning_rate": 4.868598382749327e-05,
+      "loss": 0.2254,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 2.4483835697174072,
+      "learning_rate": 4.834905660377358e-05,
+      "loss": 0.2099,
+      "step": 1690
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 3.059027671813965,
+      "learning_rate": 4.801212938005391e-05,
+      "loss": 0.2054,
+      "step": 1700
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 2.2340033054351807,
+      "learning_rate": 4.7675202156334234e-05,
+      "loss": 0.1919,
+      "step": 1710
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 3.059826374053955,
+      "learning_rate": 4.733827493261456e-05,
+      "loss": 0.202,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 2.4581353664398193,
+      "learning_rate": 4.7001347708894886e-05,
+      "loss": 0.2025,
+      "step": 1730
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 3.7301676273345947,
+      "learning_rate": 4.66644204851752e-05,
+      "loss": 0.2119,
+      "step": 1740
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 2.9101502895355225,
+      "learning_rate": 4.632749326145553e-05,
+      "loss": 0.2102,
+      "step": 1750
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 1.4716150760650635,
+      "learning_rate": 4.5990566037735846e-05,
+      "loss": 0.1975,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 1.810396432876587,
+      "learning_rate": 4.5653638814016176e-05,
+      "loss": 0.1818,
+      "step": 1770
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 2.5587878227233887,
+      "learning_rate": 4.53167115902965e-05,
+      "loss": 0.205,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 2.6180355548858643,
+      "learning_rate": 4.497978436657682e-05,
+      "loss": 0.2066,
+      "step": 1790
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 3.3715500831604004,
+      "learning_rate": 4.464285714285715e-05,
+      "loss": 0.1986,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 1.9417994022369385,
+      "learning_rate": 4.4305929919137465e-05,
+      "loss": 0.18,
+      "step": 1810
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 2.6988582611083984,
+      "learning_rate": 4.3969002695417794e-05,
+      "loss": 0.1848,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 2.6839194297790527,
+      "learning_rate": 4.363207547169812e-05,
+      "loss": 0.2074,
+      "step": 1830
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 2.4248201847076416,
+      "learning_rate": 4.329514824797844e-05,
+      "loss": 0.1929,
+      "step": 1840
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 2.474905252456665,
+      "learning_rate": 4.295822102425876e-05,
+      "loss": 0.1761,
+      "step": 1850
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 1.7574677467346191,
+      "learning_rate": 4.2621293800539084e-05,
+      "loss": 0.1693,
+      "step": 1860
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 2.314537286758423,
+      "learning_rate": 4.2284366576819406e-05,
+      "loss": 0.1683,
+      "step": 1870
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 2.127652406692505,
+      "learning_rate": 4.1947439353099736e-05,
+      "loss": 0.2025,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 3.3728537559509277,
+      "learning_rate": 4.161051212938006e-05,
+      "loss": 0.1697,
+      "step": 1890
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 1.9143751859664917,
+      "learning_rate": 4.127358490566038e-05,
+      "loss": 0.1669,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 2.564091205596924,
+      "learning_rate": 4.09366576819407e-05,
+      "loss": 0.1695,
+      "step": 1910
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 3.54518985748291,
+      "learning_rate": 4.0599730458221025e-05,
+      "loss": 0.1836,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 1.8193870782852173,
+      "learning_rate": 4.026280323450135e-05,
+      "loss": 0.1701,
+      "step": 1930
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 3.027362585067749,
+      "learning_rate": 3.992587601078167e-05,
+      "loss": 0.1572,
+      "step": 1940
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 2.244178295135498,
+      "learning_rate": 3.9588948787062e-05,
+      "loss": 0.164,
+      "step": 1950
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 3.8362321853637695,
+      "learning_rate": 3.9252021563342315e-05,
+      "loss": 0.1667,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 3.5789124965667725,
+      "learning_rate": 3.8915094339622644e-05,
+      "loss": 0.1642,
+      "step": 1970
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 3.8518402576446533,
+      "learning_rate": 3.8578167115902966e-05,
+      "loss": 0.1615,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 2.248826265335083,
+      "learning_rate": 3.824123989218329e-05,
+      "loss": 0.1591,
+      "step": 1990
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 2.504639148712158,
+      "learning_rate": 3.790431266846362e-05,
+      "loss": 0.1679,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 2.4511938095092773,
+      "learning_rate": 3.7567385444743934e-05,
+      "loss": 0.1887,
+      "step": 2010
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 2.2310004234313965,
+      "learning_rate": 3.723045822102426e-05,
+      "loss": 0.1584,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 2.4412670135498047,
+      "learning_rate": 3.689353099730458e-05,
+      "loss": 0.1493,
+      "step": 2030
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 2.1844723224639893,
+      "learning_rate": 3.655660377358491e-05,
+      "loss": 0.1636,
+      "step": 2040
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 2.738367795944214,
+      "learning_rate": 3.621967654986524e-05,
+      "loss": 0.1602,
+      "step": 2050
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 2.138552188873291,
+      "learning_rate": 3.588274932614555e-05,
+      "loss": 0.159,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 1.3966834545135498,
+      "learning_rate": 3.554582210242588e-05,
+      "loss": 0.1428,
+      "step": 2070
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 2.3553030490875244,
+      "learning_rate": 3.52088948787062e-05,
+      "loss": 0.1542,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 2.553690195083618,
+      "learning_rate": 3.4871967654986526e-05,
+      "loss": 0.1433,
+      "step": 2090
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 2.555643081665039,
+      "learning_rate": 3.453504043126685e-05,
+      "loss": 0.139,
+      "step": 2100
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 2.7808148860931396,
+      "learning_rate": 3.419811320754717e-05,
+      "loss": 0.1688,
+      "step": 2110
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 3.2580206394195557,
+      "learning_rate": 3.3861185983827494e-05,
+      "loss": 0.1508,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 4.284587860107422,
+      "learning_rate": 3.3524258760107816e-05,
+      "loss": 0.1613,
+      "step": 2130
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 1.9140340089797974,
+      "learning_rate": 3.3187331536388145e-05,
+      "loss": 0.1562,
+      "step": 2140
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 2.13785719871521,
+      "learning_rate": 3.285040431266847e-05,
+      "loss": 0.1465,
+      "step": 2150
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 2.165259599685669,
+      "learning_rate": 3.251347708894879e-05,
+      "loss": 0.1254,
+      "step": 2160
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 2.14497971534729,
+      "learning_rate": 3.217654986522911e-05,
+      "loss": 0.1436,
+      "step": 2170
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 2.4378933906555176,
+      "learning_rate": 3.1839622641509435e-05,
+      "loss": 0.154,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 2.225801467895508,
+      "learning_rate": 3.150269541778976e-05,
+      "loss": 0.1485,
+      "step": 2190
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 1.9187358617782593,
+      "learning_rate": 3.1165768194070086e-05,
+      "loss": 0.1312,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 2.536228895187378,
+      "learning_rate": 3.08288409703504e-05,
+      "loss": 0.1574,
+      "step": 2210
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 1.6395888328552246,
+      "learning_rate": 3.0491913746630728e-05,
+      "loss": 0.133,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 1.9579951763153076,
+      "learning_rate": 3.0154986522911054e-05,
+      "loss": 0.139,
+      "step": 2230
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 2.6751959323883057,
+      "learning_rate": 2.9818059299191376e-05,
+      "loss": 0.1312,
+      "step": 2240
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 2.2726802825927734,
+      "learning_rate": 2.9481132075471702e-05,
+      "loss": 0.1324,
+      "step": 2250
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 1.9153738021850586,
+      "learning_rate": 2.914420485175202e-05,
+      "loss": 0.1271,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 1.5008968114852905,
+      "learning_rate": 2.8807277628032347e-05,
+      "loss": 0.1268,
+      "step": 2270
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 1.671263575553894,
+      "learning_rate": 2.847035040431267e-05,
+      "loss": 0.1108,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 2.4821507930755615,
+      "learning_rate": 2.8133423180592995e-05,
+      "loss": 0.1251,
+      "step": 2290
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 1.8863062858581543,
+      "learning_rate": 2.7796495956873314e-05,
+      "loss": 0.1224,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 2.204700469970703,
+      "learning_rate": 2.745956873315364e-05,
+      "loss": 0.1206,
+      "step": 2310
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 3.5700721740722656,
+      "learning_rate": 2.7122641509433965e-05,
+      "loss": 0.1379,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 1.4034169912338257,
+      "learning_rate": 2.6785714285714288e-05,
+      "loss": 0.1289,
+      "step": 2330
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 2.7189619541168213,
+      "learning_rate": 2.6448787061994614e-05,
+      "loss": 0.1374,
+      "step": 2340
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 1.9794065952301025,
+      "learning_rate": 2.6111859838274933e-05,
+      "loss": 0.1306,
+      "step": 2350
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 1.698568344116211,
+      "learning_rate": 2.577493261455526e-05,
+      "loss": 0.1279,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 1.7154345512390137,
+      "learning_rate": 2.5438005390835577e-05,
+      "loss": 0.1354,
+      "step": 2370
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 1.5892575979232788,
+      "learning_rate": 2.5101078167115903e-05,
+      "loss": 0.1308,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 1.742138147354126,
+      "learning_rate": 2.476415094339623e-05,
+      "loss": 0.1239,
+      "step": 2390
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 2.4784200191497803,
+      "learning_rate": 2.442722371967655e-05,
+      "loss": 0.1235,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 1.3302199840545654,
+      "learning_rate": 2.4090296495956874e-05,
+      "loss": 0.1193,
+      "step": 2410
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 2.079979658126831,
+      "learning_rate": 2.3753369272237196e-05,
+      "loss": 0.1252,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 2.201711893081665,
+      "learning_rate": 2.341644204851752e-05,
+      "loss": 0.118,
+      "step": 2430
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 1.476678729057312,
+      "learning_rate": 2.3079514824797844e-05,
+      "loss": 0.1116,
+      "step": 2440
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 1.3897534608840942,
+      "learning_rate": 2.274258760107817e-05,
+      "loss": 0.1116,
+      "step": 2450
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 2.085052728652954,
+      "learning_rate": 2.2405660377358493e-05,
+      "loss": 0.121,
+      "step": 2460
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 1.4823940992355347,
+      "learning_rate": 2.2068733153638815e-05,
+      "loss": 0.1242,
+      "step": 2470
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 1.9251364469528198,
+      "learning_rate": 2.1731805929919137e-05,
+      "loss": 0.1182,
+      "step": 2480
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 1.792622447013855,
+      "learning_rate": 2.1394878706199463e-05,
+      "loss": 0.1127,
+      "step": 2490
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 2.0481772422790527,
+      "learning_rate": 2.1057951482479785e-05,
+      "loss": 0.1137,
+      "step": 2500
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 2.2373077869415283,
+      "learning_rate": 2.0721024258760108e-05,
+      "loss": 0.1162,
+      "step": 2510
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 1.4368358850479126,
+      "learning_rate": 2.038409703504043e-05,
+      "loss": 0.1102,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 1.7859045267105103,
+      "learning_rate": 2.0047169811320756e-05,
+      "loss": 0.119,
+      "step": 2530
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 2.55025315284729,
+      "learning_rate": 1.971024258760108e-05,
+      "loss": 0.1168,
+      "step": 2540
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 1.7251120805740356,
+      "learning_rate": 1.9373315363881404e-05,
+      "loss": 0.1203,
+      "step": 2550
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 1.499519944190979,
+      "learning_rate": 1.9036388140161727e-05,
+      "loss": 0.1091,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 2.4695260524749756,
+      "learning_rate": 1.869946091644205e-05,
+      "loss": 0.1238,
+      "step": 2570
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 1.6461279392242432,
+      "learning_rate": 1.836253369272237e-05,
+      "loss": 0.1113,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 2.1009840965270996,
+      "learning_rate": 1.8025606469002694e-05,
+      "loss": 0.1153,
+      "step": 2590
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 1.6194204092025757,
+      "learning_rate": 1.768867924528302e-05,
+      "loss": 0.1248,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 1.8107424974441528,
+      "learning_rate": 1.7351752021563345e-05,
+      "loss": 0.1126,
+      "step": 2610
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 1.6274248361587524,
+      "learning_rate": 1.7014824797843668e-05,
+      "loss": 0.1119,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 1.5619142055511475,
+      "learning_rate": 1.667789757412399e-05,
+      "loss": 0.1121,
+      "step": 2630
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 2.297623634338379,
+      "learning_rate": 1.6340970350404313e-05,
+      "loss": 0.1246,
+      "step": 2640
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 1.5185441970825195,
+      "learning_rate": 1.600404312668464e-05,
+      "loss": 0.103,
+      "step": 2650
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 1.9803398847579956,
+      "learning_rate": 1.566711590296496e-05,
+      "loss": 0.1173,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 2.065286874771118,
+      "learning_rate": 1.5330188679245283e-05,
+      "loss": 0.1166,
+      "step": 2670
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 1.6951541900634766,
+      "learning_rate": 1.4993261455525606e-05,
+      "loss": 0.1036,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 1.2760865688323975,
+      "learning_rate": 1.465633423180593e-05,
+      "loss": 0.1086,
+      "step": 2690
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 2.6494252681732178,
+      "learning_rate": 1.4319407008086256e-05,
+      "loss": 0.1063,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 1.4181417226791382,
+      "learning_rate": 1.3982479784366578e-05,
+      "loss": 0.1159,
+      "step": 2710
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 1.8402090072631836,
+      "learning_rate": 1.3645552560646902e-05,
+      "loss": 0.1098,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 2.2500264644622803,
+      "learning_rate": 1.3308625336927224e-05,
+      "loss": 0.1126,
+      "step": 2730
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 1.485680103302002,
+      "learning_rate": 1.2971698113207547e-05,
+      "loss": 0.106,
+      "step": 2740
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 1.3506594896316528,
+      "learning_rate": 1.2634770889487871e-05,
+      "loss": 0.1099,
+      "step": 2750
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 1.615838885307312,
+      "learning_rate": 1.2297843665768195e-05,
+      "loss": 0.2101,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 1.8304481506347656,
+      "learning_rate": 1.1960916442048519e-05,
+      "loss": 0.1022,
+      "step": 2770
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 1.7675200700759888,
+      "learning_rate": 1.1623989218328842e-05,
+      "loss": 0.102,
+      "step": 2780
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 1.363110899925232,
+      "learning_rate": 1.1287061994609164e-05,
+      "loss": 0.1078,
+      "step": 2790
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 3.771144151687622,
+      "learning_rate": 1.0950134770889488e-05,
+      "loss": 0.1099,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 1.606154203414917,
+      "learning_rate": 1.0613207547169812e-05,
+      "loss": 0.1128,
+      "step": 2810
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 1.24965238571167,
+      "learning_rate": 1.0276280323450135e-05,
+      "loss": 0.0987,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 1.2326825857162476,
+      "learning_rate": 9.939353099730459e-06,
+      "loss": 0.1114,
+      "step": 2830
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 1.5760172605514526,
+      "learning_rate": 9.602425876010781e-06,
+      "loss": 0.1114,
+      "step": 2840
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 2.7495434284210205,
+      "learning_rate": 9.265498652291107e-06,
+      "loss": 0.1108,
+      "step": 2850
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 1.2853425741195679,
+      "learning_rate": 8.92857142857143e-06,
+      "loss": 0.1015,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 1.6944804191589355,
+      "learning_rate": 8.591644204851752e-06,
+      "loss": 0.0976,
+      "step": 2870
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 1.2788165807724,
+      "learning_rate": 8.254716981132076e-06,
+      "loss": 0.1088,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 1.9614472389221191,
+      "learning_rate": 7.9177897574124e-06,
+      "loss": 0.1057,
+      "step": 2890
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 1.3156172037124634,
+      "learning_rate": 7.580862533692723e-06,
+      "loss": 0.116,
+      "step": 2900
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 2.4010279178619385,
+      "learning_rate": 7.243935309973046e-06,
+      "loss": 0.1127,
+      "step": 2910
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 2.3485734462738037,
+      "learning_rate": 6.90700808625337e-06,
+      "loss": 0.1061,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 2.0748932361602783,
+      "learning_rate": 6.570080862533692e-06,
+      "loss": 0.1048,
+      "step": 2930
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 1.0390431880950928,
+      "learning_rate": 6.233153638814016e-06,
+      "loss": 0.1039,
+      "step": 2940
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 1.1470861434936523,
+      "learning_rate": 5.89622641509434e-06,
+      "loss": 0.1021,
+      "step": 2950
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 1.8556820154190063,
+      "learning_rate": 5.5592991913746634e-06,
+      "loss": 0.1025,
+      "step": 2960
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 1.5888488292694092,
+      "learning_rate": 5.222371967654987e-06,
+      "loss": 0.1126,
+      "step": 2970
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 1.8423320055007935,
+      "learning_rate": 4.88544474393531e-06,
+      "loss": 0.1084,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 1.38668954372406,
+      "learning_rate": 4.548517520215634e-06,
+      "loss": 0.1029,
+      "step": 2990
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 2.0503017902374268,
+      "learning_rate": 4.211590296495957e-06,
+      "loss": 0.1082,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 1.526978850364685,
+      "learning_rate": 3.8746630727762805e-06,
+      "loss": 0.0986,
+      "step": 3010
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 2.662074089050293,
+      "learning_rate": 3.5377358490566038e-06,
+      "loss": 0.1093,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 1.9316941499710083,
+      "learning_rate": 3.200808625336928e-06,
+      "loss": 0.0976,
+      "step": 3030
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 2.845834732055664,
+      "learning_rate": 2.8638814016172507e-06,
+      "loss": 0.1076,
+      "step": 3040
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 2.393538475036621,
+      "learning_rate": 2.5269541778975744e-06,
+      "loss": 0.1015,
+      "step": 3050
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 2.8218283653259277,
+      "learning_rate": 2.1900269541778976e-06,
+      "loss": 0.1197,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 1.0874052047729492,
+      "learning_rate": 1.853099730458221e-06,
+      "loss": 0.0988,
+      "step": 3070
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 1.1840276718139648,
+      "learning_rate": 1.5161725067385445e-06,
+      "loss": 0.1024,
+      "step": 3080
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 1.8606740236282349,
+      "learning_rate": 1.179245283018868e-06,
+      "loss": 0.106,
+      "step": 3090
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 1.9203367233276367,
+      "learning_rate": 8.423180592991913e-07,
+      "loss": 0.1041,
+      "step": 3100
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 1.15613853931427,
+      "learning_rate": 5.053908355795148e-07,
+      "loss": 0.114,
+      "step": 3110
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 2.514665365219116,
+      "learning_rate": 1.684636118598383e-07,
+      "loss": 0.1012,
+      "step": 3120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 3125,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3125/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1dc4d08bb7516a729cd57b78eec1fb5d196503456ad554a6f6c71d0116ebd452
+size 5304

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "gpt2-medium",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "n_special": 0,
+  "predict_special_tokens": true,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "transformers_version": "4.45.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_logs.csv

@@ -0,0 +1,314 @@
+step,training_loss,grad_norm
+10,22.4391,10.000000953674316
+20,18.5398,9.999999046325684
+30,18.8696,10.0
+40,17.9832,9.999999046325684
+50,15.0181,10.0
+60,14.2746,9.999999046325684
+70,11.9335,10.0
+80,9.6611,10.0
+90,8.2423,10.0
+100,7.0975,9.999999046325684
+110,5.2427,10.0
+120,5.0576,9.999999046325684
+130,4.3215,9.999999046325684
+140,3.7479,9.999999046325684
+150,3.5213,9.422039985656738
+160,3.2425,9.999998092651367
+170,2.8891,9.351380348205566
+180,2.783,8.307005882263184
+190,2.7527,8.963129997253418
+200,2.4355,8.570749282836914
+210,2.4091,8.063360214233398
+220,2.1608,8.824004173278809
+230,2.2534,7.373661994934082
+240,2.1537,8.307866096496582
+250,1.9916,5.856404781341553
+260,1.9904,8.64548397064209
+270,2.0503,7.271899223327637
+280,1.9192,6.368246555328369
+290,1.6597,6.038721561431885
+300,1.9203,6.260958194732666
+310,1.8547,7.62371301651001
+320,1.7249,7.491437911987305
+330,1.5601,7.197510719299316
+340,1.6026,6.928409099578857
+350,1.5981,6.700569152832031
+360,1.4863,6.20810604095459
+370,1.5054,6.265394687652588
+380,1.4892,7.249783039093018
+390,1.3346,6.698989391326904
+400,1.269,6.408270359039307
+410,1.2479,6.221065521240234
+420,1.1823,6.697783470153809
+430,1.2215,6.298666000366211
+440,1.2094,5.8949875831604
+450,1.1668,4.865890026092529
+460,1.0941,4.905447006225586
+470,1.081,5.108652591705322
+480,1.0708,4.898155212402344
+490,0.9574,4.2186455726623535
+500,1.1016,4.5638957023620605
+510,0.9938,5.078132152557373
+520,1.0042,6.263752460479736
+530,0.8717,4.613877773284912
+540,0.9252,5.016404151916504
+550,0.9297,4.927691459655762
+560,0.8883,4.506190299987793
+570,0.8302,5.3836350440979
+580,0.8401,4.673761367797852
+590,0.888,3.7030556201934814
+600,0.8483,4.952301025390625
+610,0.7925,4.4545769691467285
+620,0.7347,4.0674896240234375
+630,0.7413,6.8568034172058105
+640,0.7714,6.715765476226807
+650,0.7337,4.419723033905029
+660,0.6854,4.291261672973633
+670,0.7224,5.974247932434082
+680,0.6737,4.334232807159424
+690,0.6505,4.699735164642334
+700,0.6696,4.12949275970459
+710,0.6587,3.9896364212036133
+720,0.6189,4.438923358917236
+730,0.6249,3.4031989574432373
+740,0.6462,5.202292442321777
+750,0.5842,5.27857780456543
+760,0.5659,4.678514003753662
+770,0.5833,4.691657543182373
+780,0.5958,7.131917476654053
+790,0.6307,5.070577144622803
+800,0.6064,4.827574253082275
+810,0.5218,3.8818304538726807
+820,0.5342,6.581722736358643
+830,0.5167,9.561187744140625
+840,0.5674,6.698048114776611
+850,0.5432,5.142230987548828
+860,0.557,5.226676940917969
+870,0.4969,5.462721824645996
+880,0.5004,4.489198207855225
+890,0.4465,4.706643104553223
+900,0.4989,7.614391803741455
+910,0.4978,3.3230834007263184
+920,0.4477,4.60692834854126
+930,0.4573,4.93078088760376
+940,0.4262,3.973932981491089
+950,0.4804,3.4613964557647705
+960,0.4037,3.4361777305603027
+970,0.4436,3.738327980041504
+980,0.4489,5.539529323577881
+990,0.4346,3.828397512435913
+1000,0.4615,4.032661437988281
+1010,0.4137,3.523750066757202
+1020,0.4397,5.352044582366943
+1030,0.4411,3.491257667541504
+1040,0.3997,2.7892098426818848
+1050,0.4118,3.231407880783081
+1060,0.3631,4.61359977722168
+1070,0.399,7.804078578948975
+1080,0.4628,5.450227737426758
+1090,0.4149,4.55374002456665
+1100,0.3892,2.5150911808013916
+1110,0.3874,3.4889912605285645
+1120,0.4126,4.045688629150391
+1130,0.3564,5.061463356018066
+1140,0.4036,3.681854248046875
+1150,0.4049,4.347032070159912
+1160,0.3708,7.9803314208984375
+1170,0.376,5.374613285064697
+1180,0.371,3.0691614151000977
+1190,0.3681,4.836121082305908
+1200,0.3379,3.778252124786377
+1210,0.3627,3.3291821479797363
+1220,0.3423,3.522620916366577
+1230,0.3301,3.283186912536621
+1240,0.3468,4.433807849884033
+1250,0.3376,3.97624135017395
+1260,0.3032,3.079582929611206
+1270,0.3452,4.162789821624756
+1280,0.3351,4.1800665855407715
+1290,0.3579,4.206212520599365
+1300,0.3328,3.900646924972534
+1310,0.345,5.285961627960205
+1320,0.321,2.616771697998047
+1330,0.3188,3.762444496154785
+1340,0.3299,3.244722366333008
+1350,0.3063,3.7463319301605225
+1360,0.3218,2.2992026805877686
+1370,0.3295,3.3672423362731934
+1380,0.282,3.5933585166931152
+1390,0.3113,5.269179821014404
+1400,0.2977,2.99794340133667
+1410,0.2954,3.103388547897339
+1420,0.2779,2.7087464332580566
+1430,0.3277,4.316291809082031
+1440,0.2738,3.2305455207824707
+1450,0.2764,2.637054443359375
+1460,0.2624,2.727341890335083
+1470,0.2889,3.334686756134033
+1480,0.3006,2.98366641998291
+1490,0.2628,3.3115627765655518
+1500,0.2607,4.2003583908081055
+1510,0.2869,6.368921756744385
+1520,0.2757,6.977168083190918
+1530,0.2545,2.90818452835083
+1540,0.2632,3.735295534133911
+1550,0.2347,3.361546039581299
+1560,0.2601,4.228140830993652
+1570,0.2604,4.071884632110596
+1580,0.2353,2.7090647220611572
+1590,0.2311,5.145593166351318
+1600,0.2466,3.904247522354126
+1610,0.267,1.8991984128952026
+1620,0.2648,3.3171005249023438
+1630,0.2293,2.8795769214630127
+1640,0.2243,3.4193594455718994
+1650,0.2339,3.470912218093872
+1660,0.2346,4.2241926193237305
+1670,0.2162,2.4578678607940674
+1680,0.2254,2.8263041973114014
+1690,0.2099,2.4483835697174072
+1700,0.2054,3.059027671813965
+1710,0.1919,2.2340033054351807
+1720,0.202,3.059826374053955
+1730,0.2025,2.4581353664398193
+1740,0.2119,3.7301676273345947
+1750,0.2102,2.9101502895355225
+1760,0.1975,1.4716150760650635
+1770,0.1818,1.810396432876587
+1780,0.205,2.5587878227233887
+1790,0.2066,2.6180355548858643
+1800,0.1986,3.3715500831604004
+1810,0.18,1.9417994022369385
+1820,0.1848,2.6988582611083984
+1830,0.2074,2.6839194297790527
+1840,0.1929,2.4248201847076416
+1850,0.1761,2.474905252456665
+1860,0.1693,1.7574677467346191
+1870,0.1683,2.314537286758423
+1880,0.2025,2.127652406692505
+1890,0.1697,3.3728537559509277
+1900,0.1669,1.9143751859664917
+1910,0.1695,2.564091205596924
+1920,0.1836,3.54518985748291
+1930,0.1701,1.8193870782852173
+1940,0.1572,3.027362585067749
+1950,0.164,2.244178295135498
+1960,0.1667,3.8362321853637695
+1970,0.1642,3.5789124965667725
+1980,0.1615,3.8518402576446533
+1990,0.1591,2.248826265335083
+2000,0.1679,2.504639148712158
+2010,0.1887,2.4511938095092773
+2020,0.1584,2.2310004234313965
+2030,0.1493,2.4412670135498047
+2040,0.1636,2.1844723224639893
+2050,0.1602,2.738367795944214
+2060,0.159,2.138552188873291
+2070,0.1428,1.3966834545135498
+2080,0.1542,2.3553030490875244
+2090,0.1433,2.553690195083618
+2100,0.139,2.555643081665039
+2110,0.1688,2.7808148860931396
+2120,0.1508,3.2580206394195557
+2130,0.1613,4.284587860107422
+2140,0.1562,1.9140340089797974
+2150,0.1465,2.13785719871521
+2160,0.1254,2.165259599685669
+2170,0.1436,2.14497971534729
+2180,0.154,2.4378933906555176
+2190,0.1485,2.225801467895508
+2200,0.1312,1.9187358617782593
+2210,0.1574,2.536228895187378
+2220,0.133,1.6395888328552246
+2230,0.139,1.9579951763153076
+2240,0.1312,2.6751959323883057
+2250,0.1324,2.2726802825927734
+2260,0.1271,1.9153738021850586
+2270,0.1268,1.5008968114852905
+2280,0.1108,1.671263575553894
+2290,0.1251,2.4821507930755615
+2300,0.1224,1.8863062858581543
+2310,0.1206,2.204700469970703
+2320,0.1379,3.5700721740722656
+2330,0.1289,1.4034169912338257
+2340,0.1374,2.7189619541168213
+2350,0.1306,1.9794065952301025
+2360,0.1279,1.698568344116211
+2370,0.1354,1.7154345512390137
+2380,0.1308,1.5892575979232788
+2390,0.1239,1.742138147354126
+2400,0.1235,2.4784200191497803
+2410,0.1193,1.3302199840545654
+2420,0.1252,2.079979658126831
+2430,0.118,2.201711893081665
+2440,0.1116,1.476678729057312
+2450,0.1116,1.3897534608840942
+2460,0.121,2.085052728652954
+2470,0.1242,1.4823940992355347
+2480,0.1182,1.9251364469528198
+2490,0.1127,1.792622447013855
+2500,0.1137,2.0481772422790527
+2510,0.1162,2.2373077869415283
+2520,0.1102,1.4368358850479126
+2530,0.119,1.7859045267105103
+2540,0.1168,2.55025315284729
+2550,0.1203,1.7251120805740356
+2560,0.1091,1.499519944190979
+2570,0.1238,2.4695260524749756
+2580,0.1113,1.6461279392242432
+2590,0.1153,2.1009840965270996
+2600,0.1248,1.6194204092025757
+2610,0.1126,1.8107424974441528
+2620,0.1119,1.6274248361587524
+2630,0.1121,1.5619142055511475
+2640,0.1246,2.297623634338379
+2650,0.103,1.5185441970825195
+2660,0.1173,1.9803398847579956
+2670,0.1166,2.065286874771118
+2680,0.1036,1.6951541900634766
+2690,0.1086,1.2760865688323975
+2700,0.1063,2.6494252681732178
+2710,0.1159,1.4181417226791382
+2720,0.1098,1.8402090072631836
+2730,0.1126,2.2500264644622803
+2740,0.106,1.485680103302002
+2750,0.1099,1.3506594896316528
+2760,0.2101,1.615838885307312
+2770,0.1022,1.8304481506347656
+2780,0.102,1.7675200700759888
+2790,0.1078,1.363110899925232
+2800,0.1099,3.771144151687622
+2810,0.1128,1.606154203414917
+2820,0.0987,1.24965238571167
+2830,0.1114,1.2326825857162476
+2840,0.1114,1.5760172605514526
+2850,0.1108,2.7495434284210205
+2860,0.1015,1.2853425741195679
+2870,0.0976,1.6944804191589355
+2880,0.1088,1.2788165807724
+2890,0.1057,1.9614472389221191
+2900,0.116,1.3156172037124634
+2910,0.1127,2.4010279178619385
+2920,0.1061,2.3485734462738037
+2930,0.1048,2.0748932361602783
+2940,0.1039,1.0390431880950928
+2950,0.1021,1.1470861434936523
+2960,0.1025,1.8556820154190063
+2970,0.1126,1.5888488292694092
+2980,0.1084,1.8423320055007935
+2990,0.1029,1.38668954372406
+3000,0.1082,2.0503017902374268
+3010,0.0986,1.526978850364685
+3020,0.1093,2.662074089050293
+3030,0.0976,1.9316941499710083
+3040,0.1076,2.845834732055664
+3050,0.1015,2.393538475036621
+3060,0.1197,2.8218283653259277
+3070,0.0988,1.0874052047729492
+3080,0.1024,1.1840276718139648
+3090,0.106,1.8606740236282349
+3100,0.1041,1.9203367233276367
+3110,0.114,1.15613853931427
+3120,0.1012,2.514665365219116
+3125,nan,nan