Upload folder using huggingface_hub

README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66292a1403d0bc2511bfda4cc1a981d2f0e3a43e9dc24a699d9acbbd3e8c7d82
+size 25172088

checkpoint-3125/README.md

@@ -0,0 +1,202 @@
+---
+base_model: gpt2-medium
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.13.2

checkpoint-3125/adapter_config.json

@@ -0,0 +1,31 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": {
+    "base_model_class": "GPT2LMHeadModel",
+    "parent_library": "transformers.models.gpt2.modeling_gpt2"
+  },
+  "base_model_name_or_path": "gpt2-medium",
+  "bias": "none",
+  "fan_in_fan_out": true,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 128,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "c_attn"
+  ],
+  "task_type": null,
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-3125/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66292a1403d0bc2511bfda4cc1a981d2f0e3a43e9dc24a699d9acbbd3e8c7d82
+size 25172088

checkpoint-3125/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1c1f81515b0e252f5c7a4b8546046b582904357f2f2a50140a617f23ab20d8f
+size 50372538

checkpoint-3125/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:feded292b067a53a2aeb0e2a23dd8fd5fd080c27efc2767fe4b430da7e5f7d6f
+size 14244

checkpoint-3125/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e42e57efa8b95889e87232fe6ea0805483ed6c8a2d35bb60230715b624ff5bda
+size 1064

checkpoint-3125/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-3125/tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

checkpoint-3125/trainer_state.json

@@ -0,0 +1,2217 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 3125,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0032,
+      "grad_norm": 10.0,
+      "learning_rate": 6.369426751592357e-06,
+      "loss": 24.9631,
+      "step": 10
+    },
+    {
+      "epoch": 0.0064,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 1.2738853503184714e-05,
+      "loss": 20.8404,
+      "step": 20
+    },
+    {
+      "epoch": 0.0096,
+      "grad_norm": 10.0,
+      "learning_rate": 1.910828025477707e-05,
+      "loss": 18.8159,
+      "step": 30
+    },
+    {
+      "epoch": 0.0128,
+      "grad_norm": 10.0,
+      "learning_rate": 2.5477707006369428e-05,
+      "loss": 20.6325,
+      "step": 40
+    },
+    {
+      "epoch": 0.016,
+      "grad_norm": 10.0,
+      "learning_rate": 3.184713375796178e-05,
+      "loss": 17.4707,
+      "step": 50
+    },
+    {
+      "epoch": 0.0192,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 3.821656050955414e-05,
+      "loss": 15.7267,
+      "step": 60
+    },
+    {
+      "epoch": 0.0224,
+      "grad_norm": 10.000000953674316,
+      "learning_rate": 4.45859872611465e-05,
+      "loss": 14.4446,
+      "step": 70
+    },
+    {
+      "epoch": 0.0256,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 5.0955414012738855e-05,
+      "loss": 12.1937,
+      "step": 80
+    },
+    {
+      "epoch": 0.0288,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 5.732484076433121e-05,
+      "loss": 9.5339,
+      "step": 90
+    },
+    {
+      "epoch": 0.032,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 6.369426751592356e-05,
+      "loss": 6.8862,
+      "step": 100
+    },
+    {
+      "epoch": 0.0352,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 7.006369426751592e-05,
+      "loss": 6.4802,
+      "step": 110
+    },
+    {
+      "epoch": 0.0384,
+      "grad_norm": 9.999998092651367,
+      "learning_rate": 7.643312101910829e-05,
+      "loss": 4.1698,
+      "step": 120
+    },
+    {
+      "epoch": 0.0416,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 8.280254777070065e-05,
+      "loss": 3.6582,
+      "step": 130
+    },
+    {
+      "epoch": 0.0448,
+      "grad_norm": 8.310874938964844,
+      "learning_rate": 8.9171974522293e-05,
+      "loss": 3.0036,
+      "step": 140
+    },
+    {
+      "epoch": 0.048,
+      "grad_norm": 8.249914169311523,
+      "learning_rate": 9.554140127388536e-05,
+      "loss": 2.5991,
+      "step": 150
+    },
+    {
+      "epoch": 0.0512,
+      "grad_norm": 6.633706092834473,
+      "learning_rate": 9.989892183288411e-05,
+      "loss": 2.0956,
+      "step": 160
+    },
+    {
+      "epoch": 0.0544,
+      "grad_norm": 4.844052791595459,
+      "learning_rate": 9.956199460916442e-05,
+      "loss": 1.8054,
+      "step": 170
+    },
+    {
+      "epoch": 0.0576,
+      "grad_norm": 5.696422100067139,
+      "learning_rate": 9.922506738544474e-05,
+      "loss": 1.5541,
+      "step": 180
+    },
+    {
+      "epoch": 0.0608,
+      "grad_norm": 5.249777793884277,
+      "learning_rate": 9.888814016172507e-05,
+      "loss": 1.5431,
+      "step": 190
+    },
+    {
+      "epoch": 0.064,
+      "grad_norm": 4.493651866912842,
+      "learning_rate": 9.85512129380054e-05,
+      "loss": 1.5084,
+      "step": 200
+    },
+    {
+      "epoch": 0.0672,
+      "grad_norm": 3.875426769256592,
+      "learning_rate": 9.821428571428572e-05,
+      "loss": 1.3255,
+      "step": 210
+    },
+    {
+      "epoch": 0.0704,
+      "grad_norm": 3.985583782196045,
+      "learning_rate": 9.787735849056603e-05,
+      "loss": 1.368,
+      "step": 220
+    },
+    {
+      "epoch": 0.0736,
+      "grad_norm": 3.1382219791412354,
+      "learning_rate": 9.754043126684636e-05,
+      "loss": 1.2237,
+      "step": 230
+    },
+    {
+      "epoch": 0.0768,
+      "grad_norm": 3.6973352432250977,
+      "learning_rate": 9.720350404312669e-05,
+      "loss": 1.1944,
+      "step": 240
+    },
+    {
+      "epoch": 0.08,
+      "grad_norm": 4.306434154510498,
+      "learning_rate": 9.686657681940702e-05,
+      "loss": 1.1278,
+      "step": 250
+    },
+    {
+      "epoch": 0.0832,
+      "grad_norm": 4.24944543838501,
+      "learning_rate": 9.652964959568734e-05,
+      "loss": 1.1202,
+      "step": 260
+    },
+    {
+      "epoch": 0.0864,
+      "grad_norm": 4.12693977355957,
+      "learning_rate": 9.619272237196765e-05,
+      "loss": 1.0113,
+      "step": 270
+    },
+    {
+      "epoch": 0.0896,
+      "grad_norm": 3.7358357906341553,
+      "learning_rate": 9.585579514824798e-05,
+      "loss": 1.0587,
+      "step": 280
+    },
+    {
+      "epoch": 0.0928,
+      "grad_norm": 4.148690223693848,
+      "learning_rate": 9.551886792452831e-05,
+      "loss": 0.9658,
+      "step": 290
+    },
+    {
+      "epoch": 0.096,
+      "grad_norm": 4.104267120361328,
+      "learning_rate": 9.518194070080863e-05,
+      "loss": 1.0358,
+      "step": 300
+    },
+    {
+      "epoch": 0.0992,
+      "grad_norm": 3.458801746368408,
+      "learning_rate": 9.484501347708896e-05,
+      "loss": 0.9948,
+      "step": 310
+    },
+    {
+      "epoch": 0.1024,
+      "grad_norm": 2.915538787841797,
+      "learning_rate": 9.450808625336927e-05,
+      "loss": 0.8794,
+      "step": 320
+    },
+    {
+      "epoch": 0.1056,
+      "grad_norm": 3.209948778152466,
+      "learning_rate": 9.41711590296496e-05,
+      "loss": 0.9644,
+      "step": 330
+    },
+    {
+      "epoch": 0.1088,
+      "grad_norm": 3.135376214981079,
+      "learning_rate": 9.383423180592993e-05,
+      "loss": 0.9274,
+      "step": 340
+    },
+    {
+      "epoch": 0.112,
+      "grad_norm": 3.4967312812805176,
+      "learning_rate": 9.349730458221025e-05,
+      "loss": 0.9099,
+      "step": 350
+    },
+    {
+      "epoch": 0.1152,
+      "grad_norm": 3.2876088619232178,
+      "learning_rate": 9.316037735849057e-05,
+      "loss": 0.8661,
+      "step": 360
+    },
+    {
+      "epoch": 0.1184,
+      "grad_norm": 3.465593099594116,
+      "learning_rate": 9.282345013477089e-05,
+      "loss": 0.8903,
+      "step": 370
+    },
+    {
+      "epoch": 0.1216,
+      "grad_norm": 3.491227388381958,
+      "learning_rate": 9.248652291105122e-05,
+      "loss": 0.8406,
+      "step": 380
+    },
+    {
+      "epoch": 0.1248,
+      "grad_norm": 3.153066396713257,
+      "learning_rate": 9.214959568733154e-05,
+      "loss": 0.8485,
+      "step": 390
+    },
+    {
+      "epoch": 0.128,
+      "grad_norm": 4.715076446533203,
+      "learning_rate": 9.181266846361186e-05,
+      "loss": 0.8795,
+      "step": 400
+    },
+    {
+      "epoch": 0.1312,
+      "grad_norm": 3.2449190616607666,
+      "learning_rate": 9.14757412398922e-05,
+      "loss": 0.8124,
+      "step": 410
+    },
+    {
+      "epoch": 0.1344,
+      "grad_norm": 3.496378183364868,
+      "learning_rate": 9.113881401617251e-05,
+      "loss": 0.7733,
+      "step": 420
+    },
+    {
+      "epoch": 0.1376,
+      "grad_norm": 3.975687265396118,
+      "learning_rate": 9.080188679245284e-05,
+      "loss": 0.7438,
+      "step": 430
+    },
+    {
+      "epoch": 0.1408,
+      "grad_norm": 4.541060447692871,
+      "learning_rate": 9.046495956873315e-05,
+      "loss": 0.7557,
+      "step": 440
+    },
+    {
+      "epoch": 0.144,
+      "grad_norm": 3.7180514335632324,
+      "learning_rate": 9.012803234501348e-05,
+      "loss": 0.7744,
+      "step": 450
+    },
+    {
+      "epoch": 0.1472,
+      "grad_norm": 4.477784156799316,
+      "learning_rate": 8.979110512129381e-05,
+      "loss": 0.7894,
+      "step": 460
+    },
+    {
+      "epoch": 0.1504,
+      "grad_norm": 6.005988121032715,
+      "learning_rate": 8.945417789757413e-05,
+      "loss": 0.664,
+      "step": 470
+    },
+    {
+      "epoch": 0.1536,
+      "grad_norm": 3.6489133834838867,
+      "learning_rate": 8.911725067385444e-05,
+      "loss": 0.6678,
+      "step": 480
+    },
+    {
+      "epoch": 0.1568,
+      "grad_norm": 4.066476821899414,
+      "learning_rate": 8.878032345013477e-05,
+      "loss": 0.6436,
+      "step": 490
+    },
+    {
+      "epoch": 0.16,
+      "grad_norm": 2.9249703884124756,
+      "learning_rate": 8.84433962264151e-05,
+      "loss": 0.6698,
+      "step": 500
+    },
+    {
+      "epoch": 0.1632,
+      "grad_norm": 3.470515489578247,
+      "learning_rate": 8.810646900269543e-05,
+      "loss": 0.6342,
+      "step": 510
+    },
+    {
+      "epoch": 0.1664,
+      "grad_norm": 3.9201626777648926,
+      "learning_rate": 8.776954177897575e-05,
+      "loss": 0.6154,
+      "step": 520
+    },
+    {
+      "epoch": 0.1696,
+      "grad_norm": 3.164458990097046,
+      "learning_rate": 8.743261455525606e-05,
+      "loss": 0.5471,
+      "step": 530
+    },
+    {
+      "epoch": 0.1728,
+      "grad_norm": 8.615708351135254,
+      "learning_rate": 8.709568733153639e-05,
+      "loss": 0.6391,
+      "step": 540
+    },
+    {
+      "epoch": 0.176,
+      "grad_norm": 2.548628330230713,
+      "learning_rate": 8.675876010781672e-05,
+      "loss": 0.5768,
+      "step": 550
+    },
+    {
+      "epoch": 0.1792,
+      "grad_norm": 5.320557594299316,
+      "learning_rate": 8.642183288409704e-05,
+      "loss": 0.5361,
+      "step": 560
+    },
+    {
+      "epoch": 0.1824,
+      "grad_norm": 5.885339736938477,
+      "learning_rate": 8.608490566037735e-05,
+      "loss": 0.542,
+      "step": 570
+    },
+    {
+      "epoch": 0.1856,
+      "grad_norm": 3.641576051712036,
+      "learning_rate": 8.574797843665768e-05,
+      "loss": 0.572,
+      "step": 580
+    },
+    {
+      "epoch": 0.1888,
+      "grad_norm": 4.401009559631348,
+      "learning_rate": 8.541105121293801e-05,
+      "loss": 0.5329,
+      "step": 590
+    },
+    {
+      "epoch": 0.192,
+      "grad_norm": 3.0472371578216553,
+      "learning_rate": 8.507412398921834e-05,
+      "loss": 0.5004,
+      "step": 600
+    },
+    {
+      "epoch": 0.1952,
+      "grad_norm": 4.5183491706848145,
+      "learning_rate": 8.473719676549866e-05,
+      "loss": 0.4432,
+      "step": 610
+    },
+    {
+      "epoch": 0.1984,
+      "grad_norm": 4.483486175537109,
+      "learning_rate": 8.440026954177897e-05,
+      "loss": 0.438,
+      "step": 620
+    },
+    {
+      "epoch": 0.2016,
+      "grad_norm": 3.5213429927825928,
+      "learning_rate": 8.40633423180593e-05,
+      "loss": 0.4345,
+      "step": 630
+    },
+    {
+      "epoch": 0.2048,
+      "grad_norm": 2.9145050048828125,
+      "learning_rate": 8.372641509433963e-05,
+      "loss": 0.391,
+      "step": 640
+    },
+    {
+      "epoch": 0.208,
+      "grad_norm": 2.7155418395996094,
+      "learning_rate": 8.338948787061996e-05,
+      "loss": 0.3555,
+      "step": 650
+    },
+    {
+      "epoch": 0.2112,
+      "grad_norm": 2.8801558017730713,
+      "learning_rate": 8.305256064690027e-05,
+      "loss": 0.3709,
+      "step": 660
+    },
+    {
+      "epoch": 0.2144,
+      "grad_norm": 3.217880964279175,
+      "learning_rate": 8.271563342318059e-05,
+      "loss": 0.3194,
+      "step": 670
+    },
+    {
+      "epoch": 0.2176,
+      "grad_norm": 3.9972708225250244,
+      "learning_rate": 8.237870619946092e-05,
+      "loss": 0.3249,
+      "step": 680
+    },
+    {
+      "epoch": 0.2208,
+      "grad_norm": 2.758613109588623,
+      "learning_rate": 8.204177897574125e-05,
+      "loss": 0.2929,
+      "step": 690
+    },
+    {
+      "epoch": 0.224,
+      "grad_norm": 3.3077003955841064,
+      "learning_rate": 8.170485175202158e-05,
+      "loss": 0.2982,
+      "step": 700
+    },
+    {
+      "epoch": 0.2272,
+      "grad_norm": 3.283940315246582,
+      "learning_rate": 8.136792452830189e-05,
+      "loss": 0.2758,
+      "step": 710
+    },
+    {
+      "epoch": 0.2304,
+      "grad_norm": 2.544793128967285,
+      "learning_rate": 8.103099730458221e-05,
+      "loss": 0.2943,
+      "step": 720
+    },
+    {
+      "epoch": 0.2336,
+      "grad_norm": 2.6508431434631348,
+      "learning_rate": 8.069407008086254e-05,
+      "loss": 0.2791,
+      "step": 730
+    },
+    {
+      "epoch": 0.2368,
+      "grad_norm": 3.0049996376037598,
+      "learning_rate": 8.035714285714287e-05,
+      "loss": 0.2808,
+      "step": 740
+    },
+    {
+      "epoch": 0.24,
+      "grad_norm": 3.1359918117523193,
+      "learning_rate": 8.002021563342318e-05,
+      "loss": 0.2658,
+      "step": 750
+    },
+    {
+      "epoch": 0.2432,
+      "grad_norm": 2.4908928871154785,
+      "learning_rate": 7.968328840970351e-05,
+      "loss": 0.2412,
+      "step": 760
+    },
+    {
+      "epoch": 0.2464,
+      "grad_norm": 3.4816462993621826,
+      "learning_rate": 7.934636118598383e-05,
+      "loss": 0.2347,
+      "step": 770
+    },
+    {
+      "epoch": 0.2496,
+      "grad_norm": 2.2120361328125,
+      "learning_rate": 7.900943396226416e-05,
+      "loss": 0.1969,
+      "step": 780
+    },
+    {
+      "epoch": 0.2528,
+      "grad_norm": 2.589252471923828,
+      "learning_rate": 7.867250673854449e-05,
+      "loss": 0.1985,
+      "step": 790
+    },
+    {
+      "epoch": 0.256,
+      "grad_norm": 3.6688730716705322,
+      "learning_rate": 7.83355795148248e-05,
+      "loss": 0.2202,
+      "step": 800
+    },
+    {
+      "epoch": 0.2592,
+      "grad_norm": 1.6741994619369507,
+      "learning_rate": 7.799865229110512e-05,
+      "loss": 0.1926,
+      "step": 810
+    },
+    {
+      "epoch": 0.2624,
+      "grad_norm": 1.583107352256775,
+      "learning_rate": 7.766172506738545e-05,
+      "loss": 0.1819,
+      "step": 820
+    },
+    {
+      "epoch": 0.2656,
+      "grad_norm": 3.296635389328003,
+      "learning_rate": 7.732479784366577e-05,
+      "loss": 0.1932,
+      "step": 830
+    },
+    {
+      "epoch": 0.2688,
+      "grad_norm": 2.3639001846313477,
+      "learning_rate": 7.69878706199461e-05,
+      "loss": 0.1878,
+      "step": 840
+    },
+    {
+      "epoch": 0.272,
+      "grad_norm": 1.4914251565933228,
+      "learning_rate": 7.665094339622642e-05,
+      "loss": 0.1477,
+      "step": 850
+    },
+    {
+      "epoch": 0.2752,
+      "grad_norm": 2.3486623764038086,
+      "learning_rate": 7.631401617250674e-05,
+      "loss": 0.1528,
+      "step": 860
+    },
+    {
+      "epoch": 0.2784,
+      "grad_norm": 1.2986149787902832,
+      "learning_rate": 7.597708894878706e-05,
+      "loss": 0.1699,
+      "step": 870
+    },
+    {
+      "epoch": 0.2816,
+      "grad_norm": 3.43965482711792,
+      "learning_rate": 7.56401617250674e-05,
+      "loss": 0.1868,
+      "step": 880
+    },
+    {
+      "epoch": 0.2848,
+      "grad_norm": 1.4145461320877075,
+      "learning_rate": 7.530323450134771e-05,
+      "loss": 0.1491,
+      "step": 890
+    },
+    {
+      "epoch": 0.288,
+      "grad_norm": 1.6416748762130737,
+      "learning_rate": 7.496630727762804e-05,
+      "loss": 0.1712,
+      "step": 900
+    },
+    {
+      "epoch": 0.2912,
+      "grad_norm": 2.666991949081421,
+      "learning_rate": 7.462938005390835e-05,
+      "loss": 0.257,
+      "step": 910
+    },
+    {
+      "epoch": 0.2944,
+      "grad_norm": 9.999999046325684,
+      "learning_rate": 7.429245283018868e-05,
+      "loss": 0.1845,
+      "step": 920
+    },
+    {
+      "epoch": 0.2976,
+      "grad_norm": 1.8368594646453857,
+      "learning_rate": 7.395552560646901e-05,
+      "loss": 0.1405,
+      "step": 930
+    },
+    {
+      "epoch": 0.3008,
+      "grad_norm": 2.364957571029663,
+      "learning_rate": 7.361859838274933e-05,
+      "loss": 0.1718,
+      "step": 940
+    },
+    {
+      "epoch": 0.304,
+      "grad_norm": 1.2922320365905762,
+      "learning_rate": 7.328167115902966e-05,
+      "loss": 0.1523,
+      "step": 950
+    },
+    {
+      "epoch": 0.3072,
+      "grad_norm": 2.202463150024414,
+      "learning_rate": 7.294474393530997e-05,
+      "loss": 0.1563,
+      "step": 960
+    },
+    {
+      "epoch": 0.3104,
+      "grad_norm": 3.0725717544555664,
+      "learning_rate": 7.26078167115903e-05,
+      "loss": 0.1351,
+      "step": 970
+    },
+    {
+      "epoch": 0.3136,
+      "grad_norm": 1.8276102542877197,
+      "learning_rate": 7.227088948787062e-05,
+      "loss": 0.1392,
+      "step": 980
+    },
+    {
+      "epoch": 0.3168,
+      "grad_norm": 2.689838409423828,
+      "learning_rate": 7.193396226415095e-05,
+      "loss": 0.1276,
+      "step": 990
+    },
+    {
+      "epoch": 0.32,
+      "grad_norm": 1.680222511291504,
+      "learning_rate": 7.159703504043128e-05,
+      "loss": 0.1284,
+      "step": 1000
+    },
+    {
+      "epoch": 0.3232,
+      "grad_norm": 1.0360759496688843,
+      "learning_rate": 7.126010781671159e-05,
+      "loss": 0.1239,
+      "step": 1010
+    },
+    {
+      "epoch": 0.3264,
+      "grad_norm": 1.915683388710022,
+      "learning_rate": 7.092318059299192e-05,
+      "loss": 0.1361,
+      "step": 1020
+    },
+    {
+      "epoch": 0.3296,
+      "grad_norm": 1.8662254810333252,
+      "learning_rate": 7.058625336927224e-05,
+      "loss": 0.1408,
+      "step": 1030
+    },
+    {
+      "epoch": 0.3328,
+      "grad_norm": 1.5222748517990112,
+      "learning_rate": 7.024932614555257e-05,
+      "loss": 0.1047,
+      "step": 1040
+    },
+    {
+      "epoch": 0.336,
+      "grad_norm": 1.9447592496871948,
+      "learning_rate": 6.99123989218329e-05,
+      "loss": 0.1252,
+      "step": 1050
+    },
+    {
+      "epoch": 0.3392,
+      "grad_norm": 1.8480818271636963,
+      "learning_rate": 6.957547169811321e-05,
+      "loss": 0.1198,
+      "step": 1060
+    },
+    {
+      "epoch": 0.3424,
+      "grad_norm": 1.8532341718673706,
+      "learning_rate": 6.923854447439353e-05,
+      "loss": 0.1147,
+      "step": 1070
+    },
+    {
+      "epoch": 0.3456,
+      "grad_norm": 1.458263874053955,
+      "learning_rate": 6.890161725067386e-05,
+      "loss": 0.1041,
+      "step": 1080
+    },
+    {
+      "epoch": 0.3488,
+      "grad_norm": 0.8418246507644653,
+      "learning_rate": 6.856469002695418e-05,
+      "loss": 0.0934,
+      "step": 1090
+    },
+    {
+      "epoch": 0.352,
+      "grad_norm": 1.1937845945358276,
+      "learning_rate": 6.822776280323451e-05,
+      "loss": 0.1029,
+      "step": 1100
+    },
+    {
+      "epoch": 0.3552,
+      "grad_norm": 1.4058141708374023,
+      "learning_rate": 6.789083557951483e-05,
+      "loss": 0.1057,
+      "step": 1110
+    },
+    {
+      "epoch": 0.3584,
+      "grad_norm": 2.04742169380188,
+      "learning_rate": 6.755390835579514e-05,
+      "loss": 0.1136,
+      "step": 1120
+    },
+    {
+      "epoch": 0.3616,
+      "grad_norm": 1.3528432846069336,
+      "learning_rate": 6.721698113207547e-05,
+      "loss": 0.098,
+      "step": 1130
+    },
+    {
+      "epoch": 0.3648,
+      "grad_norm": 1.3083478212356567,
+      "learning_rate": 6.68800539083558e-05,
+      "loss": 0.0982,
+      "step": 1140
+    },
+    {
+      "epoch": 0.368,
+      "grad_norm": 1.3803431987762451,
+      "learning_rate": 6.654312668463612e-05,
+      "loss": 0.1011,
+      "step": 1150
+    },
+    {
+      "epoch": 0.3712,
+      "grad_norm": 1.8763604164123535,
+      "learning_rate": 6.620619946091643e-05,
+      "loss": 0.0976,
+      "step": 1160
+    },
+    {
+      "epoch": 0.3744,
+      "grad_norm": 1.459699273109436,
+      "learning_rate": 6.586927223719676e-05,
+      "loss": 0.1021,
+      "step": 1170
+    },
+    {
+      "epoch": 0.3776,
+      "grad_norm": 2.516188621520996,
+      "learning_rate": 6.553234501347709e-05,
+      "loss": 0.094,
+      "step": 1180
+    },
+    {
+      "epoch": 0.3808,
+      "grad_norm": 2.072133779525757,
+      "learning_rate": 6.519541778975742e-05,
+      "loss": 0.0951,
+      "step": 1190
+    },
+    {
+      "epoch": 0.384,
+      "grad_norm": 1.3856562376022339,
+      "learning_rate": 6.485849056603774e-05,
+      "loss": 0.085,
+      "step": 1200
+    },
+    {
+      "epoch": 0.3872,
+      "grad_norm": 1.524794101715088,
+      "learning_rate": 6.452156334231805e-05,
+      "loss": 0.0894,
+      "step": 1210
+    },
+    {
+      "epoch": 0.3904,
+      "grad_norm": 0.8352987170219421,
+      "learning_rate": 6.418463611859838e-05,
+      "loss": 0.0813,
+      "step": 1220
+    },
+    {
+      "epoch": 0.3936,
+      "grad_norm": 1.99660062789917,
+      "learning_rate": 6.384770889487871e-05,
+      "loss": 0.0941,
+      "step": 1230
+    },
+    {
+      "epoch": 0.3968,
+      "grad_norm": 1.9948227405548096,
+      "learning_rate": 6.351078167115904e-05,
+      "loss": 0.0943,
+      "step": 1240
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 2.417297601699829,
+      "learning_rate": 6.317385444743936e-05,
+      "loss": 0.0903,
+      "step": 1250
+    },
+    {
+      "epoch": 0.4032,
+      "grad_norm": 1.255522608757019,
+      "learning_rate": 6.283692722371967e-05,
+      "loss": 0.0934,
+      "step": 1260
+    },
+    {
+      "epoch": 0.4064,
+      "grad_norm": 3.0397634506225586,
+      "learning_rate": 6.25e-05,
+      "loss": 0.0823,
+      "step": 1270
+    },
+    {
+      "epoch": 0.4096,
+      "grad_norm": 2.1224780082702637,
+      "learning_rate": 6.216307277628033e-05,
+      "loss": 0.0937,
+      "step": 1280
+    },
+    {
+      "epoch": 0.4128,
+      "grad_norm": 2.0656044483184814,
+      "learning_rate": 6.182614555256066e-05,
+      "loss": 0.0952,
+      "step": 1290
+    },
+    {
+      "epoch": 0.416,
+      "grad_norm": 1.334979772567749,
+      "learning_rate": 6.148921832884098e-05,
+      "loss": 0.0882,
+      "step": 1300
+    },
+    {
+      "epoch": 0.4192,
+      "grad_norm": 1.4411072731018066,
+      "learning_rate": 6.115229110512129e-05,
+      "loss": 0.0889,
+      "step": 1310
+    },
+    {
+      "epoch": 0.4224,
+      "grad_norm": 1.8725781440734863,
+      "learning_rate": 6.081536388140162e-05,
+      "loss": 0.0817,
+      "step": 1320
+    },
+    {
+      "epoch": 0.4256,
+      "grad_norm": 1.0607643127441406,
+      "learning_rate": 6.047843665768195e-05,
+      "loss": 0.0857,
+      "step": 1330
+    },
+    {
+      "epoch": 0.4288,
+      "grad_norm": 1.0576326847076416,
+      "learning_rate": 6.0141509433962265e-05,
+      "loss": 0.0813,
+      "step": 1340
+    },
+    {
+      "epoch": 0.432,
+      "grad_norm": 0.8751885294914246,
+      "learning_rate": 5.980458221024259e-05,
+      "loss": 0.0724,
+      "step": 1350
+    },
+    {
+      "epoch": 0.4352,
+      "grad_norm": 1.0008176565170288,
+      "learning_rate": 5.9467654986522916e-05,
+      "loss": 0.0801,
+      "step": 1360
+    },
+    {
+      "epoch": 0.4384,
+      "grad_norm": 1.0499143600463867,
+      "learning_rate": 5.913072776280324e-05,
+      "loss": 0.0827,
+      "step": 1370
+    },
+    {
+      "epoch": 0.4416,
+      "grad_norm": 1.6805224418640137,
+      "learning_rate": 5.879380053908357e-05,
+      "loss": 0.0764,
+      "step": 1380
+    },
+    {
+      "epoch": 0.4448,
+      "grad_norm": 1.0352784395217896,
+      "learning_rate": 5.8456873315363884e-05,
+      "loss": 0.0799,
+      "step": 1390
+    },
+    {
+      "epoch": 0.448,
+      "grad_norm": 1.2008415460586548,
+      "learning_rate": 5.8119946091644206e-05,
+      "loss": 0.0758,
+      "step": 1400
+    },
+    {
+      "epoch": 0.4512,
+      "grad_norm": 1.247338891029358,
+      "learning_rate": 5.7783018867924535e-05,
+      "loss": 0.0878,
+      "step": 1410
+    },
+    {
+      "epoch": 0.4544,
+      "grad_norm": 0.975530207157135,
+      "learning_rate": 5.744609164420486e-05,
+      "loss": 0.072,
+      "step": 1420
+    },
+    {
+      "epoch": 0.4576,
+      "grad_norm": 1.0029404163360596,
+      "learning_rate": 5.710916442048517e-05,
+      "loss": 0.0722,
+      "step": 1430
+    },
+    {
+      "epoch": 0.4608,
+      "grad_norm": 0.8437648415565491,
+      "learning_rate": 5.6772237196765496e-05,
+      "loss": 0.0701,
+      "step": 1440
+    },
+    {
+      "epoch": 0.464,
+      "grad_norm": 1.1050156354904175,
+      "learning_rate": 5.6435309973045825e-05,
+      "loss": 0.0695,
+      "step": 1450
+    },
+    {
+      "epoch": 0.4672,
+      "grad_norm": 0.9750869870185852,
+      "learning_rate": 5.609838274932615e-05,
+      "loss": 0.0675,
+      "step": 1460
+    },
+    {
+      "epoch": 0.4704,
+      "grad_norm": 0.8726626038551331,
+      "learning_rate": 5.5761455525606476e-05,
+      "loss": 0.0721,
+      "step": 1470
+    },
+    {
+      "epoch": 0.4736,
+      "grad_norm": 3.020092725753784,
+      "learning_rate": 5.542452830188679e-05,
+      "loss": 0.0687,
+      "step": 1480
+    },
+    {
+      "epoch": 0.4768,
+      "grad_norm": 1.3524830341339111,
+      "learning_rate": 5.5087601078167114e-05,
+      "loss": 0.0721,
+      "step": 1490
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 1.7134172916412354,
+      "learning_rate": 5.4750673854447444e-05,
+      "loss": 0.0727,
+      "step": 1500
+    },
+    {
+      "epoch": 0.4832,
+      "grad_norm": 1.3195223808288574,
+      "learning_rate": 5.4413746630727766e-05,
+      "loss": 0.0702,
+      "step": 1510
+    },
+    {
+      "epoch": 0.4864,
+      "grad_norm": 1.1717275381088257,
+      "learning_rate": 5.407681940700808e-05,
+      "loss": 0.0754,
+      "step": 1520
+    },
+    {
+      "epoch": 0.4896,
+      "grad_norm": 0.9258642792701721,
+      "learning_rate": 5.373989218328841e-05,
+      "loss": 0.0596,
+      "step": 1530
+    },
+    {
+      "epoch": 0.4928,
+      "grad_norm": 1.4086549282073975,
+      "learning_rate": 5.340296495956873e-05,
+      "loss": 0.0631,
+      "step": 1540
+    },
+    {
+      "epoch": 0.496,
+      "grad_norm": 1.211665153503418,
+      "learning_rate": 5.306603773584906e-05,
+      "loss": 0.0748,
+      "step": 1550
+    },
+    {
+      "epoch": 0.4992,
+      "grad_norm": 1.8159215450286865,
+      "learning_rate": 5.2729110512129385e-05,
+      "loss": 0.0696,
+      "step": 1560
+    },
+    {
+      "epoch": 0.5024,
+      "grad_norm": 1.5430232286453247,
+      "learning_rate": 5.23921832884097e-05,
+      "loss": 0.0629,
+      "step": 1570
+    },
+    {
+      "epoch": 0.5056,
+      "grad_norm": 0.971611499786377,
+      "learning_rate": 5.205525606469003e-05,
+      "loss": 0.0647,
+      "step": 1580
+    },
+    {
+      "epoch": 0.5088,
+      "grad_norm": 2.795301675796509,
+      "learning_rate": 5.171832884097035e-05,
+      "loss": 0.0689,
+      "step": 1590
+    },
+    {
+      "epoch": 0.512,
+      "grad_norm": 1.0469309091567993,
+      "learning_rate": 5.138140161725068e-05,
+      "loss": 0.0649,
+      "step": 1600
+    },
+    {
+      "epoch": 0.5152,
+      "grad_norm": 1.5577256679534912,
+      "learning_rate": 5.1044474393531e-05,
+      "loss": 0.0619,
+      "step": 1610
+    },
+    {
+      "epoch": 0.5184,
+      "grad_norm": 1.732717752456665,
+      "learning_rate": 5.070754716981132e-05,
+      "loss": 0.0622,
+      "step": 1620
+    },
+    {
+      "epoch": 0.5216,
+      "grad_norm": 1.4120196104049683,
+      "learning_rate": 5.037061994609165e-05,
+      "loss": 0.0672,
+      "step": 1630
+    },
+    {
+      "epoch": 0.5248,
+      "grad_norm": 1.2306411266326904,
+      "learning_rate": 5.003369272237197e-05,
+      "loss": 0.064,
+      "step": 1640
+    },
+    {
+      "epoch": 0.528,
+      "grad_norm": 2.0534772872924805,
+      "learning_rate": 4.969676549865229e-05,
+      "loss": 0.0664,
+      "step": 1650
+    },
+    {
+      "epoch": 0.5312,
+      "grad_norm": 1.562451720237732,
+      "learning_rate": 4.9359838274932616e-05,
+      "loss": 0.0665,
+      "step": 1660
+    },
+    {
+      "epoch": 0.5344,
+      "grad_norm": 1.1336791515350342,
+      "learning_rate": 4.902291105121294e-05,
+      "loss": 0.0561,
+      "step": 1670
+    },
+    {
+      "epoch": 0.5376,
+      "grad_norm": 0.9556323885917664,
+      "learning_rate": 4.868598382749327e-05,
+      "loss": 0.0579,
+      "step": 1680
+    },
+    {
+      "epoch": 0.5408,
+      "grad_norm": 1.3072831630706787,
+      "learning_rate": 4.834905660377358e-05,
+      "loss": 0.0591,
+      "step": 1690
+    },
+    {
+      "epoch": 0.544,
+      "grad_norm": 0.9365288019180298,
+      "learning_rate": 4.801212938005391e-05,
+      "loss": 0.0615,
+      "step": 1700
+    },
+    {
+      "epoch": 0.5472,
+      "grad_norm": 0.8340426683425903,
+      "learning_rate": 4.7675202156334234e-05,
+      "loss": 0.0613,
+      "step": 1710
+    },
+    {
+      "epoch": 0.5504,
+      "grad_norm": 1.1264219284057617,
+      "learning_rate": 4.733827493261456e-05,
+      "loss": 0.0569,
+      "step": 1720
+    },
+    {
+      "epoch": 0.5536,
+      "grad_norm": 1.1999742984771729,
+      "learning_rate": 4.7001347708894886e-05,
+      "loss": 0.0593,
+      "step": 1730
+    },
+    {
+      "epoch": 0.5568,
+      "grad_norm": 1.1102709770202637,
+      "learning_rate": 4.66644204851752e-05,
+      "loss": 0.0613,
+      "step": 1740
+    },
+    {
+      "epoch": 0.56,
+      "grad_norm": 1.0732330083847046,
+      "learning_rate": 4.632749326145553e-05,
+      "loss": 0.056,
+      "step": 1750
+    },
+    {
+      "epoch": 0.5632,
+      "grad_norm": 1.3392184972763062,
+      "learning_rate": 4.5990566037735846e-05,
+      "loss": 0.0546,
+      "step": 1760
+    },
+    {
+      "epoch": 0.5664,
+      "grad_norm": 1.005621314048767,
+      "learning_rate": 4.5653638814016176e-05,
+      "loss": 0.0589,
+      "step": 1770
+    },
+    {
+      "epoch": 0.5696,
+      "grad_norm": 1.4382975101470947,
+      "learning_rate": 4.53167115902965e-05,
+      "loss": 0.0597,
+      "step": 1780
+    },
+    {
+      "epoch": 0.5728,
+      "grad_norm": 1.4868929386138916,
+      "learning_rate": 4.497978436657682e-05,
+      "loss": 0.0566,
+      "step": 1790
+    },
+    {
+      "epoch": 0.576,
+      "grad_norm": 0.7314358353614807,
+      "learning_rate": 4.464285714285715e-05,
+      "loss": 0.0497,
+      "step": 1800
+    },
+    {
+      "epoch": 0.5792,
+      "grad_norm": 0.8784840106964111,
+      "learning_rate": 4.4305929919137465e-05,
+      "loss": 0.0517,
+      "step": 1810
+    },
+    {
+      "epoch": 0.5824,
+      "grad_norm": 2.5038092136383057,
+      "learning_rate": 4.3969002695417794e-05,
+      "loss": 0.0628,
+      "step": 1820
+    },
+    {
+      "epoch": 0.5856,
+      "grad_norm": 0.8334175944328308,
+      "learning_rate": 4.363207547169812e-05,
+      "loss": 0.0532,
+      "step": 1830
+    },
+    {
+      "epoch": 0.5888,
+      "grad_norm": 0.9410456418991089,
+      "learning_rate": 4.329514824797844e-05,
+      "loss": 0.0508,
+      "step": 1840
+    },
+    {
+      "epoch": 0.592,
+      "grad_norm": 0.974979817867279,
+      "learning_rate": 4.295822102425876e-05,
+      "loss": 0.0581,
+      "step": 1850
+    },
+    {
+      "epoch": 0.5952,
+      "grad_norm": 0.8645688891410828,
+      "learning_rate": 4.2621293800539084e-05,
+      "loss": 0.0541,
+      "step": 1860
+    },
+    {
+      "epoch": 0.5984,
+      "grad_norm": 0.647721529006958,
+      "learning_rate": 4.2284366576819406e-05,
+      "loss": 0.05,
+      "step": 1870
+    },
+    {
+      "epoch": 0.6016,
+      "grad_norm": 0.8814899325370789,
+      "learning_rate": 4.1947439353099736e-05,
+      "loss": 0.0504,
+      "step": 1880
+    },
+    {
+      "epoch": 0.6048,
+      "grad_norm": 0.835896372795105,
+      "learning_rate": 4.161051212938006e-05,
+      "loss": 0.06,
+      "step": 1890
+    },
+    {
+      "epoch": 0.608,
+      "grad_norm": 1.4004669189453125,
+      "learning_rate": 4.127358490566038e-05,
+      "loss": 0.0514,
+      "step": 1900
+    },
+    {
+      "epoch": 0.6112,
+      "grad_norm": 1.7096471786499023,
+      "learning_rate": 4.09366576819407e-05,
+      "loss": 0.0554,
+      "step": 1910
+    },
+    {
+      "epoch": 0.6144,
+      "grad_norm": 0.7163381576538086,
+      "learning_rate": 4.0599730458221025e-05,
+      "loss": 0.0531,
+      "step": 1920
+    },
+    {
+      "epoch": 0.6176,
+      "grad_norm": 0.8033366799354553,
+      "learning_rate": 4.026280323450135e-05,
+      "loss": 0.0535,
+      "step": 1930
+    },
+    {
+      "epoch": 0.6208,
+      "grad_norm": 1.36751389503479,
+      "learning_rate": 3.992587601078167e-05,
+      "loss": 0.0537,
+      "step": 1940
+    },
+    {
+      "epoch": 0.624,
+      "grad_norm": 0.6608594655990601,
+      "learning_rate": 3.9588948787062e-05,
+      "loss": 0.0508,
+      "step": 1950
+    },
+    {
+      "epoch": 0.6272,
+      "grad_norm": 1.4556374549865723,
+      "learning_rate": 3.9252021563342315e-05,
+      "loss": 0.054,
+      "step": 1960
+    },
+    {
+      "epoch": 0.6304,
+      "grad_norm": 1.026332139968872,
+      "learning_rate": 3.8915094339622644e-05,
+      "loss": 0.0557,
+      "step": 1970
+    },
+    {
+      "epoch": 0.6336,
+      "grad_norm": 1.2365747690200806,
+      "learning_rate": 3.8578167115902966e-05,
+      "loss": 0.0511,
+      "step": 1980
+    },
+    {
+      "epoch": 0.6368,
+      "grad_norm": 1.0087261199951172,
+      "learning_rate": 3.824123989218329e-05,
+      "loss": 0.0426,
+      "step": 1990
+    },
+    {
+      "epoch": 0.64,
+      "grad_norm": 0.8160260915756226,
+      "learning_rate": 3.790431266846362e-05,
+      "loss": 0.0452,
+      "step": 2000
+    },
+    {
+      "epoch": 0.6432,
+      "grad_norm": 1.6539990901947021,
+      "learning_rate": 3.7567385444743934e-05,
+      "loss": 0.0489,
+      "step": 2010
+    },
+    {
+      "epoch": 0.6464,
+      "grad_norm": 0.5857284069061279,
+      "learning_rate": 3.723045822102426e-05,
+      "loss": 0.0447,
+      "step": 2020
+    },
+    {
+      "epoch": 0.6496,
+      "grad_norm": 0.8850499987602234,
+      "learning_rate": 3.689353099730458e-05,
+      "loss": 0.0481,
+      "step": 2030
+    },
+    {
+      "epoch": 0.6528,
+      "grad_norm": 0.9914937019348145,
+      "learning_rate": 3.655660377358491e-05,
+      "loss": 0.0502,
+      "step": 2040
+    },
+    {
+      "epoch": 0.656,
+      "grad_norm": 0.9070343971252441,
+      "learning_rate": 3.621967654986524e-05,
+      "loss": 0.0471,
+      "step": 2050
+    },
+    {
+      "epoch": 0.6592,
+      "grad_norm": 0.9179836511611938,
+      "learning_rate": 3.588274932614555e-05,
+      "loss": 0.0436,
+      "step": 2060
+    },
+    {
+      "epoch": 0.6624,
+      "grad_norm": 0.7364044785499573,
+      "learning_rate": 3.554582210242588e-05,
+      "loss": 0.0486,
+      "step": 2070
+    },
+    {
+      "epoch": 0.6656,
+      "grad_norm": 0.6488219499588013,
+      "learning_rate": 3.52088948787062e-05,
+      "loss": 0.046,
+      "step": 2080
+    },
+    {
+      "epoch": 0.6688,
+      "grad_norm": 1.5407613515853882,
+      "learning_rate": 3.4871967654986526e-05,
+      "loss": 0.0494,
+      "step": 2090
+    },
+    {
+      "epoch": 0.672,
+      "grad_norm": 2.3234798908233643,
+      "learning_rate": 3.453504043126685e-05,
+      "loss": 0.0457,
+      "step": 2100
+    },
+    {
+      "epoch": 0.6752,
+      "grad_norm": 1.3606553077697754,
+      "learning_rate": 3.419811320754717e-05,
+      "loss": 0.0461,
+      "step": 2110
+    },
+    {
+      "epoch": 0.6784,
+      "grad_norm": 1.338086724281311,
+      "learning_rate": 3.3861185983827494e-05,
+      "loss": 0.0412,
+      "step": 2120
+    },
+    {
+      "epoch": 0.6816,
+      "grad_norm": 0.8436578512191772,
+      "learning_rate": 3.3524258760107816e-05,
+      "loss": 0.0487,
+      "step": 2130
+    },
+    {
+      "epoch": 0.6848,
+      "grad_norm": 0.9895771145820618,
+      "learning_rate": 3.3187331536388145e-05,
+      "loss": 0.0477,
+      "step": 2140
+    },
+    {
+      "epoch": 0.688,
+      "grad_norm": 1.4392690658569336,
+      "learning_rate": 3.285040431266847e-05,
+      "loss": 0.0381,
+      "step": 2150
+    },
+    {
+      "epoch": 0.6912,
+      "grad_norm": 0.8517897129058838,
+      "learning_rate": 3.251347708894879e-05,
+      "loss": 0.0513,
+      "step": 2160
+    },
+    {
+      "epoch": 0.6944,
+      "grad_norm": 0.765490710735321,
+      "learning_rate": 3.217654986522911e-05,
+      "loss": 0.0471,
+      "step": 2170
+    },
+    {
+      "epoch": 0.6976,
+      "grad_norm": 0.95233154296875,
+      "learning_rate": 3.1839622641509435e-05,
+      "loss": 0.0435,
+      "step": 2180
+    },
+    {
+      "epoch": 0.7008,
+      "grad_norm": 0.9749875068664551,
+      "learning_rate": 3.150269541778976e-05,
+      "loss": 0.0443,
+      "step": 2190
+    },
+    {
+      "epoch": 0.704,
+      "grad_norm": 1.3162479400634766,
+      "learning_rate": 3.1165768194070086e-05,
+      "loss": 0.0419,
+      "step": 2200
+    },
+    {
+      "epoch": 0.7072,
+      "grad_norm": 0.9644461870193481,
+      "learning_rate": 3.08288409703504e-05,
+      "loss": 0.0443,
+      "step": 2210
+    },
+    {
+      "epoch": 0.7104,
+      "grad_norm": 1.3393887281417847,
+      "learning_rate": 3.0491913746630728e-05,
+      "loss": 0.0462,
+      "step": 2220
+    },
+    {
+      "epoch": 0.7136,
+      "grad_norm": 0.9995754361152649,
+      "learning_rate": 3.0154986522911054e-05,
+      "loss": 0.0443,
+      "step": 2230
+    },
+    {
+      "epoch": 0.7168,
+      "grad_norm": 1.211478590965271,
+      "learning_rate": 2.9818059299191376e-05,
+      "loss": 0.0462,
+      "step": 2240
+    },
+    {
+      "epoch": 0.72,
+      "grad_norm": 0.6780794858932495,
+      "learning_rate": 2.9481132075471702e-05,
+      "loss": 0.0426,
+      "step": 2250
+    },
+    {
+      "epoch": 0.7232,
+      "grad_norm": 0.8342995047569275,
+      "learning_rate": 2.914420485175202e-05,
+      "loss": 0.0423,
+      "step": 2260
+    },
+    {
+      "epoch": 0.7264,
+      "grad_norm": 1.2715864181518555,
+      "learning_rate": 2.8807277628032347e-05,
+      "loss": 0.0449,
+      "step": 2270
+    },
+    {
+      "epoch": 0.7296,
+      "grad_norm": 0.8093573451042175,
+      "learning_rate": 2.847035040431267e-05,
+      "loss": 0.0465,
+      "step": 2280
+    },
+    {
+      "epoch": 0.7328,
+      "grad_norm": 1.1649224758148193,
+      "learning_rate": 2.8133423180592995e-05,
+      "loss": 0.0488,
+      "step": 2290
+    },
+    {
+      "epoch": 0.736,
+      "grad_norm": 1.2314190864562988,
+      "learning_rate": 2.7796495956873314e-05,
+      "loss": 0.0475,
+      "step": 2300
+    },
+    {
+      "epoch": 0.7392,
+      "grad_norm": 0.9177021980285645,
+      "learning_rate": 2.745956873315364e-05,
+      "loss": 0.047,
+      "step": 2310
+    },
+    {
+      "epoch": 0.7424,
+      "grad_norm": 1.1093103885650635,
+      "learning_rate": 2.7122641509433965e-05,
+      "loss": 0.0417,
+      "step": 2320
+    },
+    {
+      "epoch": 0.7456,
+      "grad_norm": 0.7009869813919067,
+      "learning_rate": 2.6785714285714288e-05,
+      "loss": 0.0416,
+      "step": 2330
+    },
+    {
+      "epoch": 0.7488,
+      "grad_norm": 1.0457608699798584,
+      "learning_rate": 2.6448787061994614e-05,
+      "loss": 0.0471,
+      "step": 2340
+    },
+    {
+      "epoch": 0.752,
+      "grad_norm": 0.8647756576538086,
+      "learning_rate": 2.6111859838274933e-05,
+      "loss": 0.0479,
+      "step": 2350
+    },
+    {
+      "epoch": 0.7552,
+      "grad_norm": 0.8496474027633667,
+      "learning_rate": 2.577493261455526e-05,
+      "loss": 0.0388,
+      "step": 2360
+    },
+    {
+      "epoch": 0.7584,
+      "grad_norm": 0.8979694843292236,
+      "learning_rate": 2.5438005390835577e-05,
+      "loss": 0.0428,
+      "step": 2370
+    },
+    {
+      "epoch": 0.7616,
+      "grad_norm": 1.0336112976074219,
+      "learning_rate": 2.5101078167115903e-05,
+      "loss": 0.04,
+      "step": 2380
+    },
+    {
+      "epoch": 0.7648,
+      "grad_norm": 0.8268937468528748,
+      "learning_rate": 2.476415094339623e-05,
+      "loss": 0.0412,
+      "step": 2390
+    },
+    {
+      "epoch": 0.768,
+      "grad_norm": 0.759469747543335,
+      "learning_rate": 2.442722371967655e-05,
+      "loss": 0.0415,
+      "step": 2400
+    },
+    {
+      "epoch": 0.7712,
+      "grad_norm": 0.9482242465019226,
+      "learning_rate": 2.4090296495956874e-05,
+      "loss": 0.0392,
+      "step": 2410
+    },
+    {
+      "epoch": 0.7744,
+      "grad_norm": 0.8537123799324036,
+      "learning_rate": 2.3753369272237196e-05,
+      "loss": 0.0472,
+      "step": 2420
+    },
+    {
+      "epoch": 0.7776,
+      "grad_norm": 0.6745234727859497,
+      "learning_rate": 2.341644204851752e-05,
+      "loss": 0.0449,
+      "step": 2430
+    },
+    {
+      "epoch": 0.7808,
+      "grad_norm": 0.9895679950714111,
+      "learning_rate": 2.3079514824797844e-05,
+      "loss": 0.0419,
+      "step": 2440
+    },
+    {
+      "epoch": 0.784,
+      "grad_norm": 4.300079822540283,
+      "learning_rate": 2.274258760107817e-05,
+      "loss": 0.0851,
+      "step": 2450
+    },
+    {
+      "epoch": 0.7872,
+      "grad_norm": 0.8699926137924194,
+      "learning_rate": 2.2405660377358493e-05,
+      "loss": 0.0445,
+      "step": 2460
+    },
+    {
+      "epoch": 0.7904,
+      "grad_norm": 0.7995278239250183,
+      "learning_rate": 2.2068733153638815e-05,
+      "loss": 0.0404,
+      "step": 2470
+    },
+    {
+      "epoch": 0.7936,
+      "grad_norm": 1.3382374048233032,
+      "learning_rate": 2.1731805929919137e-05,
+      "loss": 0.0399,
+      "step": 2480
+    },
+    {
+      "epoch": 0.7968,
+      "grad_norm": 0.7148540616035461,
+      "learning_rate": 2.1394878706199463e-05,
+      "loss": 0.0412,
+      "step": 2490
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 1.0621765851974487,
+      "learning_rate": 2.1057951482479785e-05,
+      "loss": 0.0372,
+      "step": 2500
+    },
+    {
+      "epoch": 0.8032,
+      "grad_norm": 1.328330636024475,
+      "learning_rate": 2.0721024258760108e-05,
+      "loss": 0.0395,
+      "step": 2510
+    },
+    {
+      "epoch": 0.8064,
+      "grad_norm": 0.7691792249679565,
+      "learning_rate": 2.038409703504043e-05,
+      "loss": 0.0423,
+      "step": 2520
+    },
+    {
+      "epoch": 0.8096,
+      "grad_norm": 0.8384915590286255,
+      "learning_rate": 2.0047169811320756e-05,
+      "loss": 0.0402,
+      "step": 2530
+    },
+    {
+      "epoch": 0.8128,
+      "grad_norm": 1.1409624814987183,
+      "learning_rate": 1.971024258760108e-05,
+      "loss": 0.0395,
+      "step": 2540
+    },
+    {
+      "epoch": 0.816,
+      "grad_norm": 0.9339775443077087,
+      "learning_rate": 1.9373315363881404e-05,
+      "loss": 0.0457,
+      "step": 2550
+    },
+    {
+      "epoch": 0.8192,
+      "grad_norm": 1.1918686628341675,
+      "learning_rate": 1.9036388140161727e-05,
+      "loss": 0.042,
+      "step": 2560
+    },
+    {
+      "epoch": 0.8224,
+      "grad_norm": 0.655660092830658,
+      "learning_rate": 1.869946091644205e-05,
+      "loss": 0.0365,
+      "step": 2570
+    },
+    {
+      "epoch": 0.8256,
+      "grad_norm": 1.1334338188171387,
+      "learning_rate": 1.836253369272237e-05,
+      "loss": 0.0468,
+      "step": 2580
+    },
+    {
+      "epoch": 0.8288,
+      "grad_norm": 1.0458253622055054,
+      "learning_rate": 1.8025606469002694e-05,
+      "loss": 0.0392,
+      "step": 2590
+    },
+    {
+      "epoch": 0.832,
+      "grad_norm": 0.9219187498092651,
+      "learning_rate": 1.768867924528302e-05,
+      "loss": 0.0381,
+      "step": 2600
+    },
+    {
+      "epoch": 0.8352,
+      "grad_norm": 0.7008917927742004,
+      "learning_rate": 1.7351752021563345e-05,
+      "loss": 0.0406,
+      "step": 2610
+    },
+    {
+      "epoch": 0.8384,
+      "grad_norm": 0.8411391973495483,
+      "learning_rate": 1.7014824797843668e-05,
+      "loss": 0.038,
+      "step": 2620
+    },
+    {
+      "epoch": 0.8416,
+      "grad_norm": 1.0414551496505737,
+      "learning_rate": 1.667789757412399e-05,
+      "loss": 0.0388,
+      "step": 2630
+    },
+    {
+      "epoch": 0.8448,
+      "grad_norm": 0.8157031536102295,
+      "learning_rate": 1.6340970350404313e-05,
+      "loss": 0.044,
+      "step": 2640
+    },
+    {
+      "epoch": 0.848,
+      "grad_norm": 1.8497180938720703,
+      "learning_rate": 1.600404312668464e-05,
+      "loss": 0.0396,
+      "step": 2650
+    },
+    {
+      "epoch": 0.8512,
+      "grad_norm": 1.7988622188568115,
+      "learning_rate": 1.566711590296496e-05,
+      "loss": 0.0388,
+      "step": 2660
+    },
+    {
+      "epoch": 0.8544,
+      "grad_norm": 1.505881667137146,
+      "learning_rate": 1.5330188679245283e-05,
+      "loss": 0.0401,
+      "step": 2670
+    },
+    {
+      "epoch": 0.8576,
+      "grad_norm": 1.3370205163955688,
+      "learning_rate": 1.4993261455525606e-05,
+      "loss": 0.039,
+      "step": 2680
+    },
+    {
+      "epoch": 0.8608,
+      "grad_norm": 0.7456148862838745,
+      "learning_rate": 1.465633423180593e-05,
+      "loss": 0.0368,
+      "step": 2690
+    },
+    {
+      "epoch": 0.864,
+      "grad_norm": 0.8330332040786743,
+      "learning_rate": 1.4319407008086256e-05,
+      "loss": 0.0402,
+      "step": 2700
+    },
+    {
+      "epoch": 0.8672,
+      "grad_norm": 0.8216585516929626,
+      "learning_rate": 1.3982479784366578e-05,
+      "loss": 0.0403,
+      "step": 2710
+    },
+    {
+      "epoch": 0.8704,
+      "grad_norm": 0.9162799715995789,
+      "learning_rate": 1.3645552560646902e-05,
+      "loss": 0.0369,
+      "step": 2720
+    },
+    {
+      "epoch": 0.8736,
+      "grad_norm": 0.9739064574241638,
+      "learning_rate": 1.3308625336927224e-05,
+      "loss": 0.0396,
+      "step": 2730
+    },
+    {
+      "epoch": 0.8768,
+      "grad_norm": 0.8707442283630371,
+      "learning_rate": 1.2971698113207547e-05,
+      "loss": 0.0368,
+      "step": 2740
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 0.678347110748291,
+      "learning_rate": 1.2634770889487871e-05,
+      "loss": 0.0351,
+      "step": 2750
+    },
+    {
+      "epoch": 0.8832,
+      "grad_norm": 1.152178168296814,
+      "learning_rate": 1.2297843665768195e-05,
+      "loss": 0.041,
+      "step": 2760
+    },
+    {
+      "epoch": 0.8864,
+      "grad_norm": 0.7693957090377808,
+      "learning_rate": 1.1960916442048519e-05,
+      "loss": 0.0378,
+      "step": 2770
+    },
+    {
+      "epoch": 0.8896,
+      "grad_norm": 1.3139463663101196,
+      "learning_rate": 1.1623989218328842e-05,
+      "loss": 0.0403,
+      "step": 2780
+    },
+    {
+      "epoch": 0.8928,
+      "grad_norm": 0.9965353012084961,
+      "learning_rate": 1.1287061994609164e-05,
+      "loss": 0.0344,
+      "step": 2790
+    },
+    {
+      "epoch": 0.896,
+      "grad_norm": 1.0216691493988037,
+      "learning_rate": 1.0950134770889488e-05,
+      "loss": 0.0396,
+      "step": 2800
+    },
+    {
+      "epoch": 0.8992,
+      "grad_norm": 0.5650272965431213,
+      "learning_rate": 1.0613207547169812e-05,
+      "loss": 0.035,
+      "step": 2810
+    },
+    {
+      "epoch": 0.9024,
+      "grad_norm": 1.3575382232666016,
+      "learning_rate": 1.0276280323450135e-05,
+      "loss": 0.038,
+      "step": 2820
+    },
+    {
+      "epoch": 0.9056,
+      "grad_norm": 0.7426192164421082,
+      "learning_rate": 9.939353099730459e-06,
+      "loss": 0.0374,
+      "step": 2830
+    },
+    {
+      "epoch": 0.9088,
+      "grad_norm": 1.6532901525497437,
+      "learning_rate": 9.602425876010781e-06,
+      "loss": 0.0385,
+      "step": 2840
+    },
+    {
+      "epoch": 0.912,
+      "grad_norm": 1.0569813251495361,
+      "learning_rate": 9.265498652291107e-06,
+      "loss": 0.0381,
+      "step": 2850
+    },
+    {
+      "epoch": 0.9152,
+      "grad_norm": 0.6354155540466309,
+      "learning_rate": 8.92857142857143e-06,
+      "loss": 0.0381,
+      "step": 2860
+    },
+    {
+      "epoch": 0.9184,
+      "grad_norm": 0.6795879602432251,
+      "learning_rate": 8.591644204851752e-06,
+      "loss": 0.04,
+      "step": 2870
+    },
+    {
+      "epoch": 0.9216,
+      "grad_norm": 1.0496159791946411,
+      "learning_rate": 8.254716981132076e-06,
+      "loss": 0.0385,
+      "step": 2880
+    },
+    {
+      "epoch": 0.9248,
+      "grad_norm": 0.6077241897583008,
+      "learning_rate": 7.9177897574124e-06,
+      "loss": 0.0392,
+      "step": 2890
+    },
+    {
+      "epoch": 0.928,
+      "grad_norm": 0.7129323482513428,
+      "learning_rate": 7.580862533692723e-06,
+      "loss": 0.0359,
+      "step": 2900
+    },
+    {
+      "epoch": 0.9312,
+      "grad_norm": 0.6670847535133362,
+      "learning_rate": 7.243935309973046e-06,
+      "loss": 0.0385,
+      "step": 2910
+    },
+    {
+      "epoch": 0.9344,
+      "grad_norm": 1.0024406909942627,
+      "learning_rate": 6.90700808625337e-06,
+      "loss": 0.0318,
+      "step": 2920
+    },
+    {
+      "epoch": 0.9376,
+      "grad_norm": 0.7983542680740356,
+      "learning_rate": 6.570080862533692e-06,
+      "loss": 0.0336,
+      "step": 2930
+    },
+    {
+      "epoch": 0.9408,
+      "grad_norm": 0.9120166301727295,
+      "learning_rate": 6.233153638814016e-06,
+      "loss": 0.0398,
+      "step": 2940
+    },
+    {
+      "epoch": 0.944,
+      "grad_norm": 0.7662246227264404,
+      "learning_rate": 5.89622641509434e-06,
+      "loss": 0.0391,
+      "step": 2950
+    },
+    {
+      "epoch": 0.9472,
+      "grad_norm": 1.092923641204834,
+      "learning_rate": 5.5592991913746634e-06,
+      "loss": 0.0358,
+      "step": 2960
+    },
+    {
+      "epoch": 0.9504,
+      "grad_norm": 0.9875261187553406,
+      "learning_rate": 5.222371967654987e-06,
+      "loss": 0.0388,
+      "step": 2970
+    },
+    {
+      "epoch": 0.9536,
+      "grad_norm": 0.6102545857429504,
+      "learning_rate": 4.88544474393531e-06,
+      "loss": 0.0327,
+      "step": 2980
+    },
+    {
+      "epoch": 0.9568,
+      "grad_norm": 1.5828185081481934,
+      "learning_rate": 4.548517520215634e-06,
+      "loss": 0.0378,
+      "step": 2990
+    },
+    {
+      "epoch": 0.96,
+      "grad_norm": 1.5497301816940308,
+      "learning_rate": 4.211590296495957e-06,
+      "loss": 0.0396,
+      "step": 3000
+    },
+    {
+      "epoch": 0.9632,
+      "grad_norm": 0.9273558259010315,
+      "learning_rate": 3.8746630727762805e-06,
+      "loss": 0.036,
+      "step": 3010
+    },
+    {
+      "epoch": 0.9664,
+      "grad_norm": 0.8762300610542297,
+      "learning_rate": 3.5377358490566038e-06,
+      "loss": 0.0336,
+      "step": 3020
+    },
+    {
+      "epoch": 0.9696,
+      "grad_norm": 1.1286474466323853,
+      "learning_rate": 3.200808625336928e-06,
+      "loss": 0.035,
+      "step": 3030
+    },
+    {
+      "epoch": 0.9728,
+      "grad_norm": 0.9490174651145935,
+      "learning_rate": 2.8638814016172507e-06,
+      "loss": 0.0339,
+      "step": 3040
+    },
+    {
+      "epoch": 0.976,
+      "grad_norm": 0.6286265850067139,
+      "learning_rate": 2.5269541778975744e-06,
+      "loss": 0.0332,
+      "step": 3050
+    },
+    {
+      "epoch": 0.9792,
+      "grad_norm": 0.9210827350616455,
+      "learning_rate": 2.1900269541778976e-06,
+      "loss": 0.0384,
+      "step": 3060
+    },
+    {
+      "epoch": 0.9824,
+      "grad_norm": 0.7867606282234192,
+      "learning_rate": 1.853099730458221e-06,
+      "loss": 0.0425,
+      "step": 3070
+    },
+    {
+      "epoch": 0.9856,
+      "grad_norm": 1.2552579641342163,
+      "learning_rate": 1.5161725067385445e-06,
+      "loss": 0.0329,
+      "step": 3080
+    },
+    {
+      "epoch": 0.9888,
+      "grad_norm": 0.4516272246837616,
+      "learning_rate": 1.179245283018868e-06,
+      "loss": 0.0348,
+      "step": 3090
+    },
+    {
+      "epoch": 0.992,
+      "grad_norm": 0.9391326904296875,
+      "learning_rate": 8.423180592991913e-07,
+      "loss": 0.039,
+      "step": 3100
+    },
+    {
+      "epoch": 0.9952,
+      "grad_norm": 1.3417290449142456,
+      "learning_rate": 5.053908355795148e-07,
+      "loss": 0.0374,
+      "step": 3110
+    },
+    {
+      "epoch": 0.9984,
+      "grad_norm": 0.6872617602348328,
+      "learning_rate": 1.684636118598383e-07,
+      "loss": 0.0321,
+      "step": 3120
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 3125,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-3125/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63dac56d38db16167e1b22287533d1716fda191bdf1dad6325013266ff8b52b5
+size 5304

config.json

@@ -0,0 +1,40 @@
+{
+  "_name_or_path": "gpt2-medium",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "n_special": 0,
+  "predict_special_tokens": true,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "transformers_version": "4.45.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<|endoftext|>",
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,22 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_logs.csv

@@ -0,0 +1,314 @@
+step,training_loss,grad_norm
+10,24.9631,10.0
+20,20.8404,9.999998092651367
+30,18.8159,10.0
+40,20.6325,10.0
+50,17.4707,10.0
+60,15.7267,9.999999046325684
+70,14.4446,10.000000953674316
+80,12.1937,9.999998092651367
+90,9.5339,9.999999046325684
+100,6.8862,9.999999046325684
+110,6.4802,9.999999046325684
+120,4.1698,9.999998092651367
+130,3.6582,9.999999046325684
+140,3.0036,8.310874938964844
+150,2.5991,8.249914169311523
+160,2.0956,6.633706092834473
+170,1.8054,4.844052791595459
+180,1.5541,5.696422100067139
+190,1.5431,5.249777793884277
+200,1.5084,4.493651866912842
+210,1.3255,3.875426769256592
+220,1.368,3.985583782196045
+230,1.2237,3.1382219791412354
+240,1.1944,3.6973352432250977
+250,1.1278,4.306434154510498
+260,1.1202,4.24944543838501
+270,1.0113,4.12693977355957
+280,1.0587,3.7358357906341553
+290,0.9658,4.148690223693848
+300,1.0358,4.104267120361328
+310,0.9948,3.458801746368408
+320,0.8794,2.915538787841797
+330,0.9644,3.209948778152466
+340,0.9274,3.135376214981079
+350,0.9099,3.4967312812805176
+360,0.8661,3.2876088619232178
+370,0.8903,3.465593099594116
+380,0.8406,3.491227388381958
+390,0.8485,3.153066396713257
+400,0.8795,4.715076446533203
+410,0.8124,3.2449190616607666
+420,0.7733,3.496378183364868
+430,0.7438,3.975687265396118
+440,0.7557,4.541060447692871
+450,0.7744,3.7180514335632324
+460,0.7894,4.477784156799316
+470,0.664,6.005988121032715
+480,0.6678,3.6489133834838867
+490,0.6436,4.066476821899414
+500,0.6698,2.9249703884124756
+510,0.6342,3.470515489578247
+520,0.6154,3.9201626777648926
+530,0.5471,3.164458990097046
+540,0.6391,8.615708351135254
+550,0.5768,2.548628330230713
+560,0.5361,5.320557594299316
+570,0.542,5.885339736938477
+580,0.572,3.641576051712036
+590,0.5329,4.401009559631348
+600,0.5004,3.0472371578216553
+610,0.4432,4.5183491706848145
+620,0.438,4.483486175537109
+630,0.4345,3.5213429927825928
+640,0.391,2.9145050048828125
+650,0.3555,2.7155418395996094
+660,0.3709,2.8801558017730713
+670,0.3194,3.217880964279175
+680,0.3249,3.9972708225250244
+690,0.2929,2.758613109588623
+700,0.2982,3.3077003955841064
+710,0.2758,3.283940315246582
+720,0.2943,2.544793128967285
+730,0.2791,2.6508431434631348
+740,0.2808,3.0049996376037598
+750,0.2658,3.1359918117523193
+760,0.2412,2.4908928871154785
+770,0.2347,3.4816462993621826
+780,0.1969,2.2120361328125
+790,0.1985,2.589252471923828
+800,0.2202,3.6688730716705322
+810,0.1926,1.6741994619369507
+820,0.1819,1.583107352256775
+830,0.1932,3.296635389328003
+840,0.1878,2.3639001846313477
+850,0.1477,1.4914251565933228
+860,0.1528,2.3486623764038086
+870,0.1699,1.2986149787902832
+880,0.1868,3.43965482711792
+890,0.1491,1.4145461320877075
+900,0.1712,1.6416748762130737
+910,0.257,2.666991949081421
+920,0.1845,9.999999046325684
+930,0.1405,1.8368594646453857
+940,0.1718,2.364957571029663
+950,0.1523,1.2922320365905762
+960,0.1563,2.202463150024414
+970,0.1351,3.0725717544555664
+980,0.1392,1.8276102542877197
+990,0.1276,2.689838409423828
+1000,0.1284,1.680222511291504
+1010,0.1239,1.0360759496688843
+1020,0.1361,1.915683388710022
+1030,0.1408,1.8662254810333252
+1040,0.1047,1.5222748517990112
+1050,0.1252,1.9447592496871948
+1060,0.1198,1.8480818271636963
+1070,0.1147,1.8532341718673706
+1080,0.1041,1.458263874053955
+1090,0.0934,0.8418246507644653
+1100,0.1029,1.1937845945358276
+1110,0.1057,1.4058141708374023
+1120,0.1136,2.04742169380188
+1130,0.098,1.3528432846069336
+1140,0.0982,1.3083478212356567
+1150,0.1011,1.3803431987762451
+1160,0.0976,1.8763604164123535
+1170,0.1021,1.459699273109436
+1180,0.094,2.516188621520996
+1190,0.0951,2.072133779525757
+1200,0.085,1.3856562376022339
+1210,0.0894,1.524794101715088
+1220,0.0813,0.8352987170219421
+1230,0.0941,1.99660062789917
+1240,0.0943,1.9948227405548096
+1250,0.0903,2.417297601699829
+1260,0.0934,1.255522608757019
+1270,0.0823,3.0397634506225586
+1280,0.0937,2.1224780082702637
+1290,0.0952,2.0656044483184814
+1300,0.0882,1.334979772567749
+1310,0.0889,1.4411072731018066
+1320,0.0817,1.8725781440734863
+1330,0.0857,1.0607643127441406
+1340,0.0813,1.0576326847076416
+1350,0.0724,0.8751885294914246
+1360,0.0801,1.0008176565170288
+1370,0.0827,1.0499143600463867
+1380,0.0764,1.6805224418640137
+1390,0.0799,1.0352784395217896
+1400,0.0758,1.2008415460586548
+1410,0.0878,1.247338891029358
+1420,0.072,0.975530207157135
+1430,0.0722,1.0029404163360596
+1440,0.0701,0.8437648415565491
+1450,0.0695,1.1050156354904175
+1460,0.0675,0.9750869870185852
+1470,0.0721,0.8726626038551331
+1480,0.0687,3.020092725753784
+1490,0.0721,1.3524830341339111
+1500,0.0727,1.7134172916412354
+1510,0.0702,1.3195223808288574
+1520,0.0754,1.1717275381088257
+1530,0.0596,0.9258642792701721
+1540,0.0631,1.4086549282073975
+1550,0.0748,1.211665153503418
+1560,0.0696,1.8159215450286865
+1570,0.0629,1.5430232286453247
+1580,0.0647,0.971611499786377
+1590,0.0689,2.795301675796509
+1600,0.0649,1.0469309091567993
+1610,0.0619,1.5577256679534912
+1620,0.0622,1.732717752456665
+1630,0.0672,1.4120196104049683
+1640,0.064,1.2306411266326904
+1650,0.0664,2.0534772872924805
+1660,0.0665,1.562451720237732
+1670,0.0561,1.1336791515350342
+1680,0.0579,0.9556323885917664
+1690,0.0591,1.3072831630706787
+1700,0.0615,0.9365288019180298
+1710,0.0613,0.8340426683425903
+1720,0.0569,1.1264219284057617
+1730,0.0593,1.1999742984771729
+1740,0.0613,1.1102709770202637
+1750,0.056,1.0732330083847046
+1760,0.0546,1.3392184972763062
+1770,0.0589,1.005621314048767
+1780,0.0597,1.4382975101470947
+1790,0.0566,1.4868929386138916
+1800,0.0497,0.7314358353614807
+1810,0.0517,0.8784840106964111
+1820,0.0628,2.5038092136383057
+1830,0.0532,0.8334175944328308
+1840,0.0508,0.9410456418991089
+1850,0.0581,0.974979817867279
+1860,0.0541,0.8645688891410828
+1870,0.05,0.647721529006958
+1880,0.0504,0.8814899325370789
+1890,0.06,0.835896372795105
+1900,0.0514,1.4004669189453125
+1910,0.0554,1.7096471786499023
+1920,0.0531,0.7163381576538086
+1930,0.0535,0.8033366799354553
+1940,0.0537,1.36751389503479
+1950,0.0508,0.6608594655990601
+1960,0.054,1.4556374549865723
+1970,0.0557,1.026332139968872
+1980,0.0511,1.2365747690200806
+1990,0.0426,1.0087261199951172
+2000,0.0452,0.8160260915756226
+2010,0.0489,1.6539990901947021
+2020,0.0447,0.5857284069061279
+2030,0.0481,0.8850499987602234
+2040,0.0502,0.9914937019348145
+2050,0.0471,0.9070343971252441
+2060,0.0436,0.9179836511611938
+2070,0.0486,0.7364044785499573
+2080,0.046,0.6488219499588013
+2090,0.0494,1.5407613515853882
+2100,0.0457,2.3234798908233643
+2110,0.0461,1.3606553077697754
+2120,0.0412,1.338086724281311
+2130,0.0487,0.8436578512191772
+2140,0.0477,0.9895771145820618
+2150,0.0381,1.4392690658569336
+2160,0.0513,0.8517897129058838
+2170,0.0471,0.765490710735321
+2180,0.0435,0.95233154296875
+2190,0.0443,0.9749875068664551
+2200,0.0419,1.3162479400634766
+2210,0.0443,0.9644461870193481
+2220,0.0462,1.3393887281417847
+2230,0.0443,0.9995754361152649
+2240,0.0462,1.211478590965271
+2250,0.0426,0.6780794858932495
+2260,0.0423,0.8342995047569275
+2270,0.0449,1.2715864181518555
+2280,0.0465,0.8093573451042175
+2290,0.0488,1.1649224758148193
+2300,0.0475,1.2314190864562988
+2310,0.047,0.9177021980285645
+2320,0.0417,1.1093103885650635
+2330,0.0416,0.7009869813919067
+2340,0.0471,1.0457608699798584
+2350,0.0479,0.8647756576538086
+2360,0.0388,0.8496474027633667
+2370,0.0428,0.8979694843292236
+2380,0.04,1.0336112976074219
+2390,0.0412,0.8268937468528748
+2400,0.0415,0.759469747543335
+2410,0.0392,0.9482242465019226
+2420,0.0472,0.8537123799324036
+2430,0.0449,0.6745234727859497
+2440,0.0419,0.9895679950714111
+2450,0.0851,4.300079822540283
+2460,0.0445,0.8699926137924194
+2470,0.0404,0.7995278239250183
+2480,0.0399,1.3382374048233032
+2490,0.0412,0.7148540616035461
+2500,0.0372,1.0621765851974487
+2510,0.0395,1.328330636024475
+2520,0.0423,0.7691792249679565
+2530,0.0402,0.8384915590286255
+2540,0.0395,1.1409624814987183
+2550,0.0457,0.9339775443077087
+2560,0.042,1.1918686628341675
+2570,0.0365,0.655660092830658
+2580,0.0468,1.1334338188171387
+2590,0.0392,1.0458253622055054
+2600,0.0381,0.9219187498092651
+2610,0.0406,0.7008917927742004
+2620,0.038,0.8411391973495483
+2630,0.0388,1.0414551496505737
+2640,0.044,0.8157031536102295
+2650,0.0396,1.8497180938720703
+2660,0.0388,1.7988622188568115
+2670,0.0401,1.505881667137146
+2680,0.039,1.3370205163955688
+2690,0.0368,0.7456148862838745
+2700,0.0402,0.8330332040786743
+2710,0.0403,0.8216585516929626
+2720,0.0369,0.9162799715995789
+2730,0.0396,0.9739064574241638
+2740,0.0368,0.8707442283630371
+2750,0.0351,0.678347110748291
+2760,0.041,1.152178168296814
+2770,0.0378,0.7693957090377808
+2780,0.0403,1.3139463663101196
+2790,0.0344,0.9965353012084961
+2800,0.0396,1.0216691493988037
+2810,0.035,0.5650272965431213
+2820,0.038,1.3575382232666016
+2830,0.0374,0.7426192164421082
+2840,0.0385,1.6532901525497437
+2850,0.0381,1.0569813251495361
+2860,0.0381,0.6354155540466309
+2870,0.04,0.6795879602432251
+2880,0.0385,1.0496159791946411
+2890,0.0392,0.6077241897583008
+2900,0.0359,0.7129323482513428
+2910,0.0385,0.6670847535133362
+2920,0.0318,1.0024406909942627
+2930,0.0336,0.7983542680740356
+2940,0.0398,0.9120166301727295
+2950,0.0391,0.7662246227264404
+2960,0.0358,1.092923641204834
+2970,0.0388,0.9875261187553406
+2980,0.0327,0.6102545857429504
+2990,0.0378,1.5828185081481934
+3000,0.0396,1.5497301816940308
+3010,0.036,0.9273558259010315
+3020,0.0336,0.8762300610542297
+3030,0.035,1.1286474466323853
+3040,0.0339,0.9490174651145935
+3050,0.0332,0.6286265850067139
+3060,0.0384,0.9210827350616455
+3070,0.0425,0.7867606282234192
+3080,0.0329,1.2552579641342163
+3090,0.0348,0.4516272246837616
+3100,0.039,0.9391326904296875
+3110,0.0374,1.3417290449142456
+3120,0.0321,0.6872617602348328
+3125,nan,nan