Instructions to use simonko912/smollm-chan-beta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use simonko912/smollm-chan-beta with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("unsloth/SmolLM2-360M")
model = PeftModel.from_pretrained(base_model, "simonko912/smollm-chan-beta")

Transformers

How to use simonko912/smollm-chan-beta with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="simonko912/smollm-chan-beta")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("simonko912/smollm-chan-beta", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use simonko912/smollm-chan-beta with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "simonko912/smollm-chan-beta"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simonko912/smollm-chan-beta",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/simonko912/smollm-chan-beta

SGLang

How to use simonko912/smollm-chan-beta with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "simonko912/smollm-chan-beta" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simonko912/smollm-chan-beta",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "simonko912/smollm-chan-beta" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simonko912/smollm-chan-beta",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Unsloth Desktop
Docker Model Runner
How to use simonko912/smollm-chan-beta with Docker Model Runner:
```
docker model run hf.co/simonko912/smollm-chan-beta
```

simonko912 commited on May 8

Commit

17e3e40

verified ·

1 Parent(s): e6e68bf

Upload folder using huggingface_hub

Browse files

Files changed (11) hide show

README.md +210 -0
adapter_config.json +52 -0
adapter_model.safetensors +3 -0
optimizer.pt +3 -0
rng_state.pth +3 -0
scaler.pt +3 -0
scheduler.pt +3 -0
tokenizer.json +0 -0
tokenizer_config.json +170 -0
trainer_state.json +1714 -0
training_args.bin +3 -0

README.md ADDED Viewed

	@@ -0,0 +1,210 @@

+---
+base_model: unsloth/SmolLM2-360M
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:unsloth/SmolLM2-360M
+- lora
+- sft
+- transformers
+- trl
+- unsloth
+---
+# Model Card for Model ID
+<!-- Provide a quick summary of what the model is/does. -->
+## Model Details
+### Model Description
+<!-- Provide a longer summary of what this model is. -->
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+### Model Sources [optional]
+<!-- Provide the basic links for the model. -->
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+## Uses
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Direct Use
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+[More Information Needed]
+### Downstream Use [optional]
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+[More Information Needed]
+### Out-of-Scope Use
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+[More Information Needed]
+## Bias, Risks, and Limitations
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+[More Information Needed]
+### Recommendations
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## How to Get Started with the Model
+Use the code below to get started with the model.
+[More Information Needed]
+## Training Details
+### Training Data
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+[More Information Needed]
+### Training Procedure
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+#### Preprocessing [optional]
+[More Information Needed]
+#### Training Hyperparameters
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+#### Speeds, Sizes, Times [optional]
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+[More Information Needed]
+## Evaluation
+<!-- This section describes the evaluation protocols and provides the results. -->
+### Testing Data, Factors & Metrics
+#### Testing Data
+<!-- This should link to a Dataset Card if possible. -->
+[More Information Needed]
+#### Factors
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+[More Information Needed]
+#### Metrics
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+[More Information Needed]
+### Results
+[More Information Needed]
+#### Summary
+## Model Examination [optional]
+<!-- Relevant interpretability work for the model goes here -->
+[More Information Needed]
+## Environmental Impact
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+## Technical Specifications [optional]
+### Model Architecture and Objective
+[More Information Needed]
+### Compute Infrastructure
+[More Information Needed]
+#### Hardware
+[More Information Needed]
+#### Software
+[More Information Needed]
+## Citation [optional]
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**BibTeX:**
+[More Information Needed]
+**APA:**
+[More Information Needed]
+## Glossary [optional]
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+[More Information Needed]
+## More Information [optional]
+[More Information Needed]
+## Model Card Authors [optional]
+[More Information Needed]
+## Model Card Contact
+[More Information Needed]
+### Framework versions
+- PEFT 0.19.1

adapter_config.json ADDED Viewed

	@@ -0,0 +1,52 @@

+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "LlamaForCausalLM",
+    "parent_library": "transformers.models.llama.modeling_llama",
+    "unsloth_fixed": true
+  },
+  "base_model_name_or_path": "unsloth/SmolLM2-360M",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "k_proj",
+    "o_proj",
+    "up_proj",
+    "q_proj",
+    "gate_proj",
+    "v_proj",
+    "down_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": true
+}

adapter_model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7bd64dfe1dd9789c56b14c0f26189b241b4b99211f54aaa7cde551c7bf34cde1
+size 138995824

optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1cd92498a3f5f64f7a831f9241f623de5d3658128238890e297e7eaea24f5fd0
+size 71083173

rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f1d565802a8e26c4e8a31328752b7a7fdc186d9401aa008e65697d0ad8c22e33
+size 14645

scaler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7da7c5085795b13d2bf0030671cbddb9f62ae43221bf1424a3830d4cf8c19012
+size 1383

scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3a52ec50569a824cb594846a2911c90a5ea2af3ea8f584079d91f7113d2f1706
+size 1465

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,170 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [],
+  "is_local": false,
+  "model_max_length": 8192,
+  "pad_token": "<|PAD_TOKEN|>",
+  "padding_side": "right",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "ï¿½",
+  "vocab_size": 49152,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "24211": {
+      "content": "ï¿½",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    "49152": {
+      "content": "<|PAD_TOKEN|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  }
+}

trainer_state.json ADDED Viewed

	@@ -0,0 +1,1714 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.1693315812218332,
+  "eval_steps": 500,
+  "global_step": 2400,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.0007055482550909716,
+      "grad_norm": 1.7742576599121094,
+      "learning_rate": 1.8e-05,
+      "loss": 3.181903266906738,
+      "step": 10
+    },
+    {
+      "epoch": 0.0014110965101819433,
+      "grad_norm": 1.3976963758468628,
+      "learning_rate": 3.8e-05,
+      "loss": 2.735004997253418,
+      "step": 20
+    },
+    {
+      "epoch": 0.002116644765272915,
+      "grad_norm": 0.6367570757865906,
+      "learning_rate": 5.8e-05,
+      "loss": 2.2967967987060547,
+      "step": 30
+    },
+    {
+      "epoch": 0.0028221930203638865,
+      "grad_norm": 0.4807576835155487,
+      "learning_rate": 7.800000000000001e-05,
+      "loss": 2.2108701705932616,
+      "step": 40
+    },
+    {
+      "epoch": 0.003527741275454858,
+      "grad_norm": 0.4114644527435303,
+      "learning_rate": 9.8e-05,
+      "loss": 2.120195770263672,
+      "step": 50
+    },
+    {
+      "epoch": 0.00423328953054583,
+      "grad_norm": 0.3620989918708801,
+      "learning_rate": 0.000118,
+      "loss": 2.0737995147705077,
+      "step": 60
+    },
+    {
+      "epoch": 0.004938837785636801,
+      "grad_norm": 0.3310359716415405,
+      "learning_rate": 0.000138,
+      "loss": 2.037908935546875,
+      "step": 70
+    },
+    {
+      "epoch": 0.005644386040727773,
+      "grad_norm": 0.34226107597351074,
+      "learning_rate": 0.00015800000000000002,
+      "loss": 2.104020881652832,
+      "step": 80
+    },
+    {
+      "epoch": 0.006349934295818744,
+      "grad_norm": 0.3665536046028137,
+      "learning_rate": 0.00017800000000000002,
+      "loss": 2.0526119232177735,
+      "step": 90
+    },
+    {
+      "epoch": 0.007055482550909716,
+      "grad_norm": 0.3421422839164734,
+      "learning_rate": 0.00019800000000000002,
+      "loss": 1.96984806060791,
+      "step": 100
+    },
+    {
+      "epoch": 0.0077610308060006876,
+      "grad_norm": 0.26802071928977966,
+      "learning_rate": 0.0001996949152542373,
+      "loss": 1.994746780395508,
+      "step": 110
+    },
+    {
+      "epoch": 0.00846657906109166,
+      "grad_norm": 0.28029438853263855,
+      "learning_rate": 0.00019935593220338983,
+      "loss": 1.8898670196533203,
+      "step": 120
+    },
+    {
+      "epoch": 0.00917212731618263,
+      "grad_norm": 0.2646600604057312,
+      "learning_rate": 0.0001990169491525424,
+      "loss": 1.973947525024414,
+      "step": 130
+    },
+    {
+      "epoch": 0.009877675571273602,
+      "grad_norm": 0.28755611181259155,
+      "learning_rate": 0.00019867796610169492,
+      "loss": 1.9899442672729493,
+      "step": 140
+    },
+    {
+      "epoch": 0.010583223826364575,
+      "grad_norm": 0.2605428099632263,
+      "learning_rate": 0.00019833898305084746,
+      "loss": 1.9590320587158203,
+      "step": 150
+    },
+    {
+      "epoch": 0.011288772081455546,
+      "grad_norm": 0.25093019008636475,
+      "learning_rate": 0.00019800000000000002,
+      "loss": 1.933210563659668,
+      "step": 160
+    },
+    {
+      "epoch": 0.011994320336546517,
+      "grad_norm": 0.25131309032440186,
+      "learning_rate": 0.00019766101694915255,
+      "loss": 1.918787384033203,
+      "step": 170
+    },
+    {
+      "epoch": 0.012699868591637489,
+      "grad_norm": 0.25882744789123535,
+      "learning_rate": 0.0001973220338983051,
+      "loss": 1.9072418212890625,
+      "step": 180
+    },
+    {
+      "epoch": 0.013405416846728462,
+      "grad_norm": 0.22492218017578125,
+      "learning_rate": 0.00019698305084745764,
+      "loss": 1.8957950592041015,
+      "step": 190
+    },
+    {
+      "epoch": 0.014110965101819433,
+      "grad_norm": 0.2546718716621399,
+      "learning_rate": 0.00019664406779661017,
+      "loss": 1.899424171447754,
+      "step": 200
+    },
+    {
+      "epoch": 0.014816513356910404,
+      "grad_norm": 0.22949501872062683,
+      "learning_rate": 0.00019630508474576273,
+      "loss": 1.9223005294799804,
+      "step": 210
+    },
+    {
+      "epoch": 0.015522061612001375,
+      "grad_norm": 0.20535190403461456,
+      "learning_rate": 0.00019596610169491527,
+      "loss": 1.8729692459106446,
+      "step": 220
+    },
+    {
+      "epoch": 0.016227609867092346,
+      "grad_norm": 0.2195505052804947,
+      "learning_rate": 0.0001956271186440678,
+      "loss": 1.9028560638427734,
+      "step": 230
+    },
+    {
+      "epoch": 0.01693315812218332,
+      "grad_norm": 0.2473132610321045,
+      "learning_rate": 0.00019528813559322033,
+      "loss": 1.892392921447754,
+      "step": 240
+    },
+    {
+      "epoch": 0.017638706377274292,
+      "grad_norm": 0.2475152164697647,
+      "learning_rate": 0.0001949491525423729,
+      "loss": 1.8456621170043945,
+      "step": 250
+    },
+    {
+      "epoch": 0.01834425463236526,
+      "grad_norm": 0.20256489515304565,
+      "learning_rate": 0.00019461016949152545,
+      "loss": 1.8871126174926758,
+      "step": 260
+    },
+    {
+      "epoch": 0.019049802887456235,
+      "grad_norm": 0.2604387402534485,
+      "learning_rate": 0.00019427118644067796,
+      "loss": 1.9076284408569335,
+      "step": 270
+    },
+    {
+      "epoch": 0.019755351142547204,
+      "grad_norm": 0.20341962575912476,
+      "learning_rate": 0.00019393220338983052,
+      "loss": 1.8776399612426757,
+      "step": 280
+    },
+    {
+      "epoch": 0.020460899397638177,
+      "grad_norm": 0.21112553775310516,
+      "learning_rate": 0.00019359322033898305,
+      "loss": 1.8270502090454102,
+      "step": 290
+    },
+    {
+      "epoch": 0.02116644765272915,
+      "grad_norm": 0.19608888030052185,
+      "learning_rate": 0.0001932542372881356,
+      "loss": 1.8436439514160157,
+      "step": 300
+    },
+    {
+      "epoch": 0.02187199590782012,
+      "grad_norm": 0.19815614819526672,
+      "learning_rate": 0.00019291525423728814,
+      "loss": 1.8614126205444337,
+      "step": 310
+    },
+    {
+      "epoch": 0.022577544162911092,
+      "grad_norm": 0.2272922843694687,
+      "learning_rate": 0.00019257627118644067,
+      "loss": 1.9396419525146484,
+      "step": 320
+    },
+    {
+      "epoch": 0.023283092418002065,
+      "grad_norm": 0.2292197346687317,
+      "learning_rate": 0.00019223728813559323,
+      "loss": 1.8869855880737305,
+      "step": 330
+    },
+    {
+      "epoch": 0.023988640673093035,
+      "grad_norm": 0.1743181198835373,
+      "learning_rate": 0.00019189830508474577,
+      "loss": 1.8280937194824218,
+      "step": 340
+    },
+    {
+      "epoch": 0.024694188928184008,
+      "grad_norm": 0.17070125043392181,
+      "learning_rate": 0.00019155932203389833,
+      "loss": 1.803072166442871,
+      "step": 350
+    },
+    {
+      "epoch": 0.025399737183274977,
+      "grad_norm": 0.17870433628559113,
+      "learning_rate": 0.00019122033898305086,
+      "loss": 1.8579156875610352,
+      "step": 360
+    },
+    {
+      "epoch": 0.02610528543836595,
+      "grad_norm": 0.18959492444992065,
+      "learning_rate": 0.0001908813559322034,
+      "loss": 1.7963617324829102,
+      "step": 370
+    },
+    {
+      "epoch": 0.026810833693456923,
+      "grad_norm": 0.1749621480703354,
+      "learning_rate": 0.00019054237288135595,
+      "loss": 1.8352516174316407,
+      "step": 380
+    },
+    {
+      "epoch": 0.027516381948547892,
+      "grad_norm": 0.17637023329734802,
+      "learning_rate": 0.00019020338983050849,
+      "loss": 1.898465919494629,
+      "step": 390
+    },
+    {
+      "epoch": 0.028221930203638865,
+      "grad_norm": 0.1701064109802246,
+      "learning_rate": 0.00018986440677966102,
+      "loss": 1.805120849609375,
+      "step": 400
+    },
+    {
+      "epoch": 0.02892747845872984,
+      "grad_norm": 0.18507348001003265,
+      "learning_rate": 0.00018952542372881358,
+      "loss": 1.8775209426879882,
+      "step": 410
+    },
+    {
+      "epoch": 0.029633026713820808,
+      "grad_norm": 0.18405862152576447,
+      "learning_rate": 0.0001891864406779661,
+      "loss": 1.8637943267822266,
+      "step": 420
+    },
+    {
+      "epoch": 0.03033857496891178,
+      "grad_norm": 0.19850201904773712,
+      "learning_rate": 0.00018884745762711867,
+      "loss": 1.7826829910278321,
+      "step": 430
+    },
+    {
+      "epoch": 0.03104412322400275,
+      "grad_norm": 0.17301882803440094,
+      "learning_rate": 0.00018850847457627118,
+      "loss": 1.8480108261108399,
+      "step": 440
+    },
+    {
+      "epoch": 0.03174967147909372,
+      "grad_norm": 0.1759461909532547,
+      "learning_rate": 0.00018816949152542374,
+      "loss": 1.8583717346191406,
+      "step": 450
+    },
+    {
+      "epoch": 0.03245521973418469,
+      "grad_norm": 0.19207057356834412,
+      "learning_rate": 0.0001878305084745763,
+      "loss": 1.80847225189209,
+      "step": 460
+    },
+    {
+      "epoch": 0.03316076798927567,
+      "grad_norm": 0.2301301658153534,
+      "learning_rate": 0.00018749152542372883,
+      "loss": 1.8642192840576173,
+      "step": 470
+    },
+    {
+      "epoch": 0.03386631624436664,
+      "grad_norm": 0.1775137037038803,
+      "learning_rate": 0.00018715254237288136,
+      "loss": 1.8408767700195312,
+      "step": 480
+    },
+    {
+      "epoch": 0.03457186449945761,
+      "grad_norm": 0.17140606045722961,
+      "learning_rate": 0.0001868135593220339,
+      "loss": 1.86510009765625,
+      "step": 490
+    },
+    {
+      "epoch": 0.035277412754548584,
+      "grad_norm": 0.18307554721832275,
+      "learning_rate": 0.00018647457627118645,
+      "loss": 1.8114852905273438,
+      "step": 500
+    },
+    {
+      "epoch": 0.035982961009639554,
+      "grad_norm": 0.16952620446681976,
+      "learning_rate": 0.000186135593220339,
+      "loss": 1.8953243255615235,
+      "step": 510
+    },
+    {
+      "epoch": 0.03668850926473052,
+      "grad_norm": 0.1644268035888672,
+      "learning_rate": 0.00018579661016949152,
+      "loss": 1.875908088684082,
+      "step": 520
+    },
+    {
+      "epoch": 0.0373940575198215,
+      "grad_norm": 0.1651521772146225,
+      "learning_rate": 0.00018545762711864408,
+      "loss": 1.8370187759399415,
+      "step": 530
+    },
+    {
+      "epoch": 0.03809960577491247,
+      "grad_norm": 0.17420922219753265,
+      "learning_rate": 0.0001851186440677966,
+      "loss": 1.805220603942871,
+      "step": 540
+    },
+    {
+      "epoch": 0.03880515403000344,
+      "grad_norm": 0.18190228939056396,
+      "learning_rate": 0.00018477966101694917,
+      "loss": 1.8316293716430665,
+      "step": 550
+    },
+    {
+      "epoch": 0.03951070228509441,
+      "grad_norm": 0.15978462994098663,
+      "learning_rate": 0.0001844406779661017,
+      "loss": 1.8022516250610352,
+      "step": 560
+    },
+    {
+      "epoch": 0.040216250540185385,
+      "grad_norm": 0.17191515862941742,
+      "learning_rate": 0.00018410169491525424,
+      "loss": 1.8303447723388673,
+      "step": 570
+    },
+    {
+      "epoch": 0.040921798795276354,
+      "grad_norm": 0.16835027933120728,
+      "learning_rate": 0.0001837627118644068,
+      "loss": 1.8877704620361329,
+      "step": 580
+    },
+    {
+      "epoch": 0.04162734705036732,
+      "grad_norm": 0.16540607810020447,
+      "learning_rate": 0.00018342372881355933,
+      "loss": 1.8321317672729491,
+      "step": 590
+    },
+    {
+      "epoch": 0.0423328953054583,
+      "grad_norm": 0.1651420295238495,
+      "learning_rate": 0.00018308474576271186,
+      "loss": 1.8570199966430665,
+      "step": 600
+    },
+    {
+      "epoch": 0.04303844356054927,
+      "grad_norm": 0.17153705656528473,
+      "learning_rate": 0.00018274576271186442,
+      "loss": 1.897837257385254,
+      "step": 610
+    },
+    {
+      "epoch": 0.04374399181564024,
+      "grad_norm": 0.1735561639070511,
+      "learning_rate": 0.00018240677966101696,
+      "loss": 1.8092647552490235,
+      "step": 620
+    },
+    {
+      "epoch": 0.044449540070731215,
+      "grad_norm": 0.1738382875919342,
+      "learning_rate": 0.00018206779661016952,
+      "loss": 1.8004777908325196,
+      "step": 630
+    },
+    {
+      "epoch": 0.045155088325822185,
+      "grad_norm": 0.16910967230796814,
+      "learning_rate": 0.00018172881355932205,
+      "loss": 1.827157211303711,
+      "step": 640
+    },
+    {
+      "epoch": 0.045860636580913154,
+      "grad_norm": 0.1659885197877884,
+      "learning_rate": 0.00018138983050847458,
+      "loss": 1.7782997131347655,
+      "step": 650
+    },
+    {
+      "epoch": 0.04656618483600413,
+      "grad_norm": 0.16636569797992706,
+      "learning_rate": 0.00018105084745762711,
+      "loss": 1.8347270965576172,
+      "step": 660
+    },
+    {
+      "epoch": 0.0472717330910951,
+      "grad_norm": 0.1639503538608551,
+      "learning_rate": 0.00018071186440677967,
+      "loss": 1.8285062789916993,
+      "step": 670
+    },
+    {
+      "epoch": 0.04797728134618607,
+      "grad_norm": 0.1655876189470291,
+      "learning_rate": 0.00018037288135593223,
+      "loss": 1.8522682189941406,
+      "step": 680
+    },
+    {
+      "epoch": 0.04868282960127704,
+      "grad_norm": 0.16768847405910492,
+      "learning_rate": 0.00018003389830508474,
+      "loss": 1.8107175827026367,
+      "step": 690
+    },
+    {
+      "epoch": 0.049388377856368015,
+      "grad_norm": 0.16892799735069275,
+      "learning_rate": 0.0001796949152542373,
+      "loss": 1.8038974761962892,
+      "step": 700
+    },
+    {
+      "epoch": 0.050093926111458985,
+      "grad_norm": 0.1631450355052948,
+      "learning_rate": 0.00017935593220338983,
+      "loss": 1.8141044616699218,
+      "step": 710
+    },
+    {
+      "epoch": 0.050799474366549954,
+      "grad_norm": 0.1683066040277481,
+      "learning_rate": 0.0001790169491525424,
+      "loss": 1.8169242858886718,
+      "step": 720
+    },
+    {
+      "epoch": 0.05150502262164093,
+      "grad_norm": 0.1518137902021408,
+      "learning_rate": 0.00017867796610169492,
+      "loss": 1.755105209350586,
+      "step": 730
+    },
+    {
+      "epoch": 0.0522105708767319,
+      "grad_norm": 0.17003647983074188,
+      "learning_rate": 0.00017833898305084746,
+      "loss": 1.757484245300293,
+      "step": 740
+    },
+    {
+      "epoch": 0.05291611913182287,
+      "grad_norm": 0.17128530144691467,
+      "learning_rate": 0.00017800000000000002,
+      "loss": 1.7809942245483399,
+      "step": 750
+    },
+    {
+      "epoch": 0.053621667386913846,
+      "grad_norm": 0.16523367166519165,
+      "learning_rate": 0.00017766101694915255,
+      "loss": 1.8317657470703126,
+      "step": 760
+    },
+    {
+      "epoch": 0.054327215642004816,
+      "grad_norm": 0.1861746609210968,
+      "learning_rate": 0.00017732203389830508,
+      "loss": 1.7801090240478517,
+      "step": 770
+    },
+    {
+      "epoch": 0.055032763897095785,
+      "grad_norm": 0.17479614913463593,
+      "learning_rate": 0.00017698305084745764,
+      "loss": 1.8350921630859376,
+      "step": 780
+    },
+    {
+      "epoch": 0.05573831215218676,
+      "grad_norm": 0.1825764924287796,
+      "learning_rate": 0.00017664406779661018,
+      "loss": 1.8303726196289063,
+      "step": 790
+    },
+    {
+      "epoch": 0.05644386040727773,
+      "grad_norm": 0.17706234753131866,
+      "learning_rate": 0.00017630508474576273,
+      "loss": 1.811711883544922,
+      "step": 800
+    },
+    {
+      "epoch": 0.0571494086623687,
+      "grad_norm": 0.18982242047786713,
+      "learning_rate": 0.00017596610169491524,
+      "loss": 1.7965097427368164,
+      "step": 810
+    },
+    {
+      "epoch": 0.05785495691745968,
+      "grad_norm": 0.1968519538640976,
+      "learning_rate": 0.0001756271186440678,
+      "loss": 1.7816627502441407,
+      "step": 820
+    },
+    {
+      "epoch": 0.058560505172550646,
+      "grad_norm": 0.16566400229930878,
+      "learning_rate": 0.00017528813559322036,
+      "loss": 1.7774896621704102,
+      "step": 830
+    },
+    {
+      "epoch": 0.059266053427641616,
+      "grad_norm": 0.17094485461711884,
+      "learning_rate": 0.0001749491525423729,
+      "loss": 1.8081693649291992,
+      "step": 840
+    },
+    {
+      "epoch": 0.059971601682732585,
+      "grad_norm": 0.16768403351306915,
+      "learning_rate": 0.00017461016949152543,
+      "loss": 1.7607406616210937,
+      "step": 850
+    },
+    {
+      "epoch": 0.06067714993782356,
+      "grad_norm": 0.17298562824726105,
+      "learning_rate": 0.00017427118644067796,
+      "loss": 1.7941802978515624,
+      "step": 860
+    },
+    {
+      "epoch": 0.06138269819291453,
+      "grad_norm": 0.16243913769721985,
+      "learning_rate": 0.00017393220338983052,
+      "loss": 1.795283889770508,
+      "step": 870
+    },
+    {
+      "epoch": 0.0620882464480055,
+      "grad_norm": 0.16944704949855804,
+      "learning_rate": 0.00017359322033898305,
+      "loss": 1.8242280960083008,
+      "step": 880
+    },
+    {
+      "epoch": 0.06279379470309647,
+      "grad_norm": 0.16967053711414337,
+      "learning_rate": 0.0001732542372881356,
+      "loss": 1.8107263565063476,
+      "step": 890
+    },
+    {
+      "epoch": 0.06349934295818745,
+      "grad_norm": 0.171393483877182,
+      "learning_rate": 0.00017291525423728814,
+      "loss": 1.755059814453125,
+      "step": 900
+    },
+    {
+      "epoch": 0.06420489121327842,
+      "grad_norm": 0.17078961431980133,
+      "learning_rate": 0.00017257627118644068,
+      "loss": 1.8068571090698242,
+      "step": 910
+    },
+    {
+      "epoch": 0.06491043946836939,
+      "grad_norm": 0.18381710350513458,
+      "learning_rate": 0.00017223728813559324,
+      "loss": 1.7932424545288086,
+      "step": 920
+    },
+    {
+      "epoch": 0.06561598772346036,
+      "grad_norm": 0.17492292821407318,
+      "learning_rate": 0.00017189830508474577,
+      "loss": 1.841135025024414,
+      "step": 930
+    },
+    {
+      "epoch": 0.06632153597855134,
+      "grad_norm": 0.16615799069404602,
+      "learning_rate": 0.0001715593220338983,
+      "loss": 1.7804277420043946,
+      "step": 940
+    },
+    {
+      "epoch": 0.0670270842336423,
+      "grad_norm": 0.1669078767299652,
+      "learning_rate": 0.00017122033898305086,
+      "loss": 1.7971153259277344,
+      "step": 950
+    },
+    {
+      "epoch": 0.06773263248873328,
+      "grad_norm": 0.16954980790615082,
+      "learning_rate": 0.0001708813559322034,
+      "loss": 1.7559749603271484,
+      "step": 960
+    },
+    {
+      "epoch": 0.06843818074382425,
+      "grad_norm": 0.18884295225143433,
+      "learning_rate": 0.00017054237288135595,
+      "loss": 1.802189826965332,
+      "step": 970
+    },
+    {
+      "epoch": 0.06914372899891522,
+      "grad_norm": 0.18105308711528778,
+      "learning_rate": 0.0001702033898305085,
+      "loss": 1.73919677734375,
+      "step": 980
+    },
+    {
+      "epoch": 0.06984927725400619,
+      "grad_norm": 0.1867796778678894,
+      "learning_rate": 0.00016986440677966102,
+      "loss": 1.7301250457763673,
+      "step": 990
+    },
+    {
+      "epoch": 0.07055482550909717,
+      "grad_norm": 0.1959427148103714,
+      "learning_rate": 0.00016952542372881358,
+      "loss": 1.7987384796142578,
+      "step": 1000
+    },
+    {
+      "epoch": 0.07126037376418813,
+      "grad_norm": 0.18582651019096375,
+      "learning_rate": 0.0001691864406779661,
+      "loss": 1.7752355575561523,
+      "step": 1010
+    },
+    {
+      "epoch": 0.07196592201927911,
+      "grad_norm": 0.18482376635074615,
+      "learning_rate": 0.00016884745762711865,
+      "loss": 1.7912906646728515,
+      "step": 1020
+    },
+    {
+      "epoch": 0.07267147027437008,
+      "grad_norm": 0.16261348128318787,
+      "learning_rate": 0.00016850847457627118,
+      "loss": 1.7841875076293945,
+      "step": 1030
+    },
+    {
+      "epoch": 0.07337701852946105,
+      "grad_norm": 0.16917714476585388,
+      "learning_rate": 0.00016816949152542374,
+      "loss": 1.742456817626953,
+      "step": 1040
+    },
+    {
+      "epoch": 0.07408256678455202,
+      "grad_norm": 0.17157341539859772,
+      "learning_rate": 0.0001678305084745763,
+      "loss": 1.7422679901123046,
+      "step": 1050
+    },
+    {
+      "epoch": 0.074788115039643,
+      "grad_norm": 0.1687636375427246,
+      "learning_rate": 0.0001674915254237288,
+      "loss": 1.7778120040893555,
+      "step": 1060
+    },
+    {
+      "epoch": 0.07549366329473396,
+      "grad_norm": 0.17122244834899902,
+      "learning_rate": 0.00016715254237288136,
+      "loss": 1.729680633544922,
+      "step": 1070
+    },
+    {
+      "epoch": 0.07619921154982494,
+      "grad_norm": 0.17000316083431244,
+      "learning_rate": 0.0001668135593220339,
+      "loss": 1.7081348419189453,
+      "step": 1080
+    },
+    {
+      "epoch": 0.0769047598049159,
+      "grad_norm": 0.17834822833538055,
+      "learning_rate": 0.00016647457627118646,
+      "loss": 1.8093843460083008,
+      "step": 1090
+    },
+    {
+      "epoch": 0.07761030806000688,
+      "grad_norm": 0.16689716279506683,
+      "learning_rate": 0.000166135593220339,
+      "loss": 1.7479795455932616,
+      "step": 1100
+    },
+    {
+      "epoch": 0.07831585631509785,
+      "grad_norm": 0.1688351333141327,
+      "learning_rate": 0.00016579661016949152,
+      "loss": 1.7417753219604493,
+      "step": 1110
+    },
+    {
+      "epoch": 0.07902140457018882,
+      "grad_norm": 0.1709015816450119,
+      "learning_rate": 0.00016545762711864408,
+      "loss": 1.770134735107422,
+      "step": 1120
+    },
+    {
+      "epoch": 0.07972695282527979,
+      "grad_norm": 0.17495828866958618,
+      "learning_rate": 0.00016511864406779661,
+      "loss": 1.7570560455322266,
+      "step": 1130
+    },
+    {
+      "epoch": 0.08043250108037077,
+      "grad_norm": 0.17297077178955078,
+      "learning_rate": 0.00016477966101694915,
+      "loss": 1.7691518783569335,
+      "step": 1140
+    },
+    {
+      "epoch": 0.08113804933546173,
+      "grad_norm": 0.16318024694919586,
+      "learning_rate": 0.0001644406779661017,
+      "loss": 1.7194955825805665,
+      "step": 1150
+    },
+    {
+      "epoch": 0.08184359759055271,
+      "grad_norm": 0.15801237523555756,
+      "learning_rate": 0.00016410169491525424,
+      "loss": 1.7584924697875977,
+      "step": 1160
+    },
+    {
+      "epoch": 0.08254914584564368,
+      "grad_norm": 0.1653916984796524,
+      "learning_rate": 0.0001637627118644068,
+      "loss": 1.7265010833740235,
+      "step": 1170
+    },
+    {
+      "epoch": 0.08325469410073465,
+      "grad_norm": 0.17348088324069977,
+      "learning_rate": 0.00016342372881355933,
+      "loss": 1.7892999649047852,
+      "step": 1180
+    },
+    {
+      "epoch": 0.08396024235582562,
+      "grad_norm": 0.16441097855567932,
+      "learning_rate": 0.00016308474576271186,
+      "loss": 1.7933538436889649,
+      "step": 1190
+    },
+    {
+      "epoch": 0.0846657906109166,
+      "grad_norm": 0.1932326704263687,
+      "learning_rate": 0.00016274576271186442,
+      "loss": 1.7072328567504882,
+      "step": 1200
+    },
+    {
+      "epoch": 0.08537133886600756,
+      "grad_norm": 0.16306975483894348,
+      "learning_rate": 0.00016240677966101696,
+      "loss": 1.7577323913574219,
+      "step": 1210
+    },
+    {
+      "epoch": 0.08607688712109854,
+      "grad_norm": 0.16868895292282104,
+      "learning_rate": 0.00016206779661016952,
+      "loss": 1.6868719100952148,
+      "step": 1220
+    },
+    {
+      "epoch": 0.08678243537618952,
+      "grad_norm": 0.1723371148109436,
+      "learning_rate": 0.00016172881355932202,
+      "loss": 1.741724395751953,
+      "step": 1230
+    },
+    {
+      "epoch": 0.08748798363128048,
+      "grad_norm": 0.18595446646213531,
+      "learning_rate": 0.00016138983050847458,
+      "loss": 1.7246334075927734,
+      "step": 1240
+    },
+    {
+      "epoch": 0.08819353188637145,
+      "grad_norm": 0.16968779265880585,
+      "learning_rate": 0.00016105084745762714,
+      "loss": 1.7884037017822265,
+      "step": 1250
+    },
+    {
+      "epoch": 0.08889908014146243,
+      "grad_norm": 0.17047403752803802,
+      "learning_rate": 0.00016071186440677968,
+      "loss": 1.7837297439575195,
+      "step": 1260
+    },
+    {
+      "epoch": 0.08960462839655339,
+      "grad_norm": 0.1983669251203537,
+      "learning_rate": 0.0001603728813559322,
+      "loss": 1.7423080444335937,
+      "step": 1270
+    },
+    {
+      "epoch": 0.09031017665164437,
+      "grad_norm": 0.18116402626037598,
+      "learning_rate": 0.00016003389830508474,
+      "loss": 1.7023443222045898,
+      "step": 1280
+    },
+    {
+      "epoch": 0.09101572490673535,
+      "grad_norm": 0.16667944192886353,
+      "learning_rate": 0.0001596949152542373,
+      "loss": 1.7319908142089844,
+      "step": 1290
+    },
+    {
+      "epoch": 0.09172127316182631,
+      "grad_norm": 0.15901318192481995,
+      "learning_rate": 0.00015935593220338983,
+      "loss": 1.7220746994018554,
+      "step": 1300
+    },
+    {
+      "epoch": 0.09242682141691728,
+      "grad_norm": 0.17020949721336365,
+      "learning_rate": 0.00015901694915254237,
+      "loss": 1.738916015625,
+      "step": 1310
+    },
+    {
+      "epoch": 0.09313236967200826,
+      "grad_norm": 0.16630448400974274,
+      "learning_rate": 0.00015867796610169493,
+      "loss": 1.7765907287597655,
+      "step": 1320
+    },
+    {
+      "epoch": 0.09383791792709922,
+      "grad_norm": 0.18998025357723236,
+      "learning_rate": 0.00015833898305084746,
+      "loss": 1.753093147277832,
+      "step": 1330
+    },
+    {
+      "epoch": 0.0945434661821902,
+      "grad_norm": 0.17990806698799133,
+      "learning_rate": 0.00015800000000000002,
+      "loss": 1.7878381729125976,
+      "step": 1340
+    },
+    {
+      "epoch": 0.09524901443728118,
+      "grad_norm": 0.1768532693386078,
+      "learning_rate": 0.00015766101694915255,
+      "loss": 1.756810188293457,
+      "step": 1350
+    },
+    {
+      "epoch": 0.09595456269237214,
+      "grad_norm": 0.17728236317634583,
+      "learning_rate": 0.00015732203389830508,
+      "loss": 1.7333850860595703,
+      "step": 1360
+    },
+    {
+      "epoch": 0.09666011094746312,
+      "grad_norm": 0.1694750040769577,
+      "learning_rate": 0.00015698305084745764,
+      "loss": 1.7704544067382812,
+      "step": 1370
+    },
+    {
+      "epoch": 0.09736565920255408,
+      "grad_norm": 0.17358361184597015,
+      "learning_rate": 0.00015664406779661018,
+      "loss": 1.746767234802246,
+      "step": 1380
+    },
+    {
+      "epoch": 0.09807120745764505,
+      "grad_norm": 0.1852710247039795,
+      "learning_rate": 0.0001563050847457627,
+      "loss": 1.7268930435180665,
+      "step": 1390
+    },
+    {
+      "epoch": 0.09877675571273603,
+      "grad_norm": 0.19318902492523193,
+      "learning_rate": 0.00015596610169491527,
+      "loss": 1.7877113342285156,
+      "step": 1400
+    },
+    {
+      "epoch": 0.099482303967827,
+      "grad_norm": 0.17854823172092438,
+      "learning_rate": 0.0001556271186440678,
+      "loss": 1.7176156997680665,
+      "step": 1410
+    },
+    {
+      "epoch": 0.10018785222291797,
+      "grad_norm": 0.18130166828632355,
+      "learning_rate": 0.00015528813559322036,
+      "loss": 1.7347988128662108,
+      "step": 1420
+    },
+    {
+      "epoch": 0.10089340047800895,
+      "grad_norm": 0.16648095846176147,
+      "learning_rate": 0.0001549491525423729,
+      "loss": 1.7390727996826172,
+      "step": 1430
+    },
+    {
+      "epoch": 0.10159894873309991,
+      "grad_norm": 0.17196723818778992,
+      "learning_rate": 0.00015461016949152543,
+      "loss": 1.7621337890625,
+      "step": 1440
+    },
+    {
+      "epoch": 0.10230449698819088,
+      "grad_norm": 0.16543926298618317,
+      "learning_rate": 0.00015427118644067796,
+      "loss": 1.7259466171264648,
+      "step": 1450
+    },
+    {
+      "epoch": 0.10301004524328186,
+      "grad_norm": 0.19407042860984802,
+      "learning_rate": 0.00015393220338983052,
+      "loss": 1.7319978713989257,
+      "step": 1460
+    },
+    {
+      "epoch": 0.10371559349837282,
+      "grad_norm": 0.1740398406982422,
+      "learning_rate": 0.00015359322033898308,
+      "loss": 1.7523180007934571,
+      "step": 1470
+    },
+    {
+      "epoch": 0.1044211417534638,
+      "grad_norm": 0.17099717259407043,
+      "learning_rate": 0.00015325423728813559,
+      "loss": 1.7171207427978517,
+      "step": 1480
+    },
+    {
+      "epoch": 0.10512669000855478,
+      "grad_norm": 0.23358525335788727,
+      "learning_rate": 0.00015291525423728815,
+      "loss": 1.7795038223266602,
+      "step": 1490
+    },
+    {
+      "epoch": 0.10583223826364574,
+      "grad_norm": 0.2014264315366745,
+      "learning_rate": 0.00015257627118644068,
+      "loss": 1.7025074005126952,
+      "step": 1500
+    },
+    {
+      "epoch": 0.10653778651873672,
+      "grad_norm": 0.18234571814537048,
+      "learning_rate": 0.00015223728813559324,
+      "loss": 1.6526134490966797,
+      "step": 1510
+    },
+    {
+      "epoch": 0.10724333477382769,
+      "grad_norm": 0.16258594393730164,
+      "learning_rate": 0.00015189830508474577,
+      "loss": 1.7437902450561524,
+      "step": 1520
+    },
+    {
+      "epoch": 0.10794888302891865,
+      "grad_norm": 0.1633520871400833,
+      "learning_rate": 0.0001515593220338983,
+      "loss": 1.724574661254883,
+      "step": 1530
+    },
+    {
+      "epoch": 0.10865443128400963,
+      "grad_norm": 0.1718776822090149,
+      "learning_rate": 0.00015122033898305086,
+      "loss": 1.7479543685913086,
+      "step": 1540
+    },
+    {
+      "epoch": 0.10935997953910061,
+      "grad_norm": 0.16803106665611267,
+      "learning_rate": 0.0001508813559322034,
+      "loss": 1.7057512283325196,
+      "step": 1550
+    },
+    {
+      "epoch": 0.11006552779419157,
+      "grad_norm": 0.17594437301158905,
+      "learning_rate": 0.00015054237288135593,
+      "loss": 1.6796890258789063,
+      "step": 1560
+    },
+    {
+      "epoch": 0.11077107604928255,
+      "grad_norm": 0.17305904626846313,
+      "learning_rate": 0.0001502033898305085,
+      "loss": 1.7236646652221679,
+      "step": 1570
+    },
+    {
+      "epoch": 0.11147662430437352,
+      "grad_norm": 0.1830369085073471,
+      "learning_rate": 0.00014986440677966102,
+      "loss": 1.7454893112182617,
+      "step": 1580
+    },
+    {
+      "epoch": 0.11218217255946449,
+      "grad_norm": 0.16984261572360992,
+      "learning_rate": 0.00014952542372881358,
+      "loss": 1.740060043334961,
+      "step": 1590
+    },
+    {
+      "epoch": 0.11288772081455546,
+      "grad_norm": 0.20283201336860657,
+      "learning_rate": 0.0001491864406779661,
+      "loss": 1.7069189071655273,
+      "step": 1600
+    },
+    {
+      "epoch": 0.11359326906964644,
+      "grad_norm": 0.1767377406358719,
+      "learning_rate": 0.00014884745762711865,
+      "loss": 1.7818607330322265,
+      "step": 1610
+    },
+    {
+      "epoch": 0.1142988173247374,
+      "grad_norm": 0.16784247756004333,
+      "learning_rate": 0.0001485084745762712,
+      "loss": 1.73642578125,
+      "step": 1620
+    },
+    {
+      "epoch": 0.11500436557982838,
+      "grad_norm": 0.16254383325576782,
+      "learning_rate": 0.00014816949152542374,
+      "loss": 1.7137287139892579,
+      "step": 1630
+    },
+    {
+      "epoch": 0.11570991383491935,
+      "grad_norm": 0.16743871569633484,
+      "learning_rate": 0.00014783050847457627,
+      "loss": 1.7191337585449218,
+      "step": 1640
+    },
+    {
+      "epoch": 0.11641546209001032,
+      "grad_norm": 0.2133561074733734,
+      "learning_rate": 0.0001474915254237288,
+      "loss": 1.751986312866211,
+      "step": 1650
+    },
+    {
+      "epoch": 0.11712101034510129,
+      "grad_norm": 0.18240249156951904,
+      "learning_rate": 0.00014715254237288137,
+      "loss": 1.6955387115478515,
+      "step": 1660
+    },
+    {
+      "epoch": 0.11782655860019227,
+      "grad_norm": 0.18928776681423187,
+      "learning_rate": 0.0001468135593220339,
+      "loss": 1.704666519165039,
+      "step": 1670
+    },
+    {
+      "epoch": 0.11853210685528323,
+      "grad_norm": 0.1897040605545044,
+      "learning_rate": 0.00014647457627118643,
+      "loss": 1.7569286346435546,
+      "step": 1680
+    },
+    {
+      "epoch": 0.11923765511037421,
+      "grad_norm": 0.17145806550979614,
+      "learning_rate": 0.000146135593220339,
+      "loss": 1.6601545333862304,
+      "step": 1690
+    },
+    {
+      "epoch": 0.11994320336546517,
+      "grad_norm": 0.2024775594472885,
+      "learning_rate": 0.00014579661016949152,
+      "loss": 1.7418800354003907,
+      "step": 1700
+    },
+    {
+      "epoch": 0.12064875162055615,
+      "grad_norm": 0.1749744862318039,
+      "learning_rate": 0.00014545762711864408,
+      "loss": 1.7143400192260743,
+      "step": 1710
+    },
+    {
+      "epoch": 0.12135429987564712,
+      "grad_norm": 0.16869990527629852,
+      "learning_rate": 0.00014511864406779662,
+      "loss": 1.6820030212402344,
+      "step": 1720
+    },
+    {
+      "epoch": 0.12205984813073809,
+      "grad_norm": 0.16981202363967896,
+      "learning_rate": 0.00014477966101694915,
+      "loss": 1.7313770294189452,
+      "step": 1730
+    },
+    {
+      "epoch": 0.12276539638582906,
+      "grad_norm": 0.17772743105888367,
+      "learning_rate": 0.0001444406779661017,
+      "loss": 1.7587583541870118,
+      "step": 1740
+    },
+    {
+      "epoch": 0.12347094464092004,
+      "grad_norm": 0.16645857691764832,
+      "learning_rate": 0.00014410169491525424,
+      "loss": 1.7862567901611328,
+      "step": 1750
+    },
+    {
+      "epoch": 0.124176492896011,
+      "grad_norm": 0.16900603473186493,
+      "learning_rate": 0.0001437627118644068,
+      "loss": 1.7517959594726562,
+      "step": 1760
+    },
+    {
+      "epoch": 0.12488204115110198,
+      "grad_norm": 0.17620205879211426,
+      "learning_rate": 0.00014342372881355933,
+      "loss": 1.7012363433837892,
+      "step": 1770
+    },
+    {
+      "epoch": 0.12558758940619294,
+      "grad_norm": 0.17485539615154266,
+      "learning_rate": 0.00014308474576271187,
+      "loss": 1.7272453308105469,
+      "step": 1780
+    },
+    {
+      "epoch": 0.12629313766128392,
+      "grad_norm": 0.18243485689163208,
+      "learning_rate": 0.00014274576271186443,
+      "loss": 1.658194351196289,
+      "step": 1790
+    },
+    {
+      "epoch": 0.1269986859163749,
+      "grad_norm": 0.16817925870418549,
+      "learning_rate": 0.00014240677966101696,
+      "loss": 1.7201351165771483,
+      "step": 1800
+    },
+    {
+      "epoch": 0.12770423417146587,
+      "grad_norm": 0.18303830921649933,
+      "learning_rate": 0.0001420677966101695,
+      "loss": 1.785972785949707,
+      "step": 1810
+    },
+    {
+      "epoch": 0.12840978242655685,
+      "grad_norm": 0.18557052314281464,
+      "learning_rate": 0.00014172881355932202,
+      "loss": 1.7419906616210938,
+      "step": 1820
+    },
+    {
+      "epoch": 0.1291153306816478,
+      "grad_norm": 0.17016908526420593,
+      "learning_rate": 0.00014138983050847458,
+      "loss": 1.7492645263671875,
+      "step": 1830
+    },
+    {
+      "epoch": 0.12982087893673877,
+      "grad_norm": 0.17999812960624695,
+      "learning_rate": 0.00014105084745762714,
+      "loss": 1.7401348114013673,
+      "step": 1840
+    },
+    {
+      "epoch": 0.13052642719182975,
+      "grad_norm": 0.17400184273719788,
+      "learning_rate": 0.00014071186440677965,
+      "loss": 1.7311624526977538,
+      "step": 1850
+    },
+    {
+      "epoch": 0.13123197544692072,
+      "grad_norm": 0.1709699034690857,
+      "learning_rate": 0.0001403728813559322,
+      "loss": 1.6810834884643555,
+      "step": 1860
+    },
+    {
+      "epoch": 0.1319375237020117,
+      "grad_norm": 0.16061733663082123,
+      "learning_rate": 0.00014003389830508474,
+      "loss": 1.7266014099121094,
+      "step": 1870
+    },
+    {
+      "epoch": 0.13264307195710268,
+      "grad_norm": 0.18397320806980133,
+      "learning_rate": 0.0001396949152542373,
+      "loss": 1.6875537872314452,
+      "step": 1880
+    },
+    {
+      "epoch": 0.13334862021219362,
+      "grad_norm": 0.17401567101478577,
+      "learning_rate": 0.00013935593220338984,
+      "loss": 1.7365072250366211,
+      "step": 1890
+    },
+    {
+      "epoch": 0.1340541684672846,
+      "grad_norm": 0.20371581614017487,
+      "learning_rate": 0.00013901694915254237,
+      "loss": 1.7328115463256837,
+      "step": 1900
+    },
+    {
+      "epoch": 0.13475971672237558,
+      "grad_norm": 0.18400566279888153,
+      "learning_rate": 0.00013867796610169493,
+      "loss": 1.7035083770751953,
+      "step": 1910
+    },
+    {
+      "epoch": 0.13546526497746655,
+      "grad_norm": 0.19648605585098267,
+      "learning_rate": 0.00013833898305084746,
+      "loss": 1.766849899291992,
+      "step": 1920
+    },
+    {
+      "epoch": 0.13617081323255753,
+      "grad_norm": 0.1778452843427658,
+      "learning_rate": 0.000138,
+      "loss": 1.6868295669555664,
+      "step": 1930
+    },
+    {
+      "epoch": 0.1368763614876485,
+      "grad_norm": 0.17346598207950592,
+      "learning_rate": 0.00013766101694915255,
+      "loss": 1.7194387435913085,
+      "step": 1940
+    },
+    {
+      "epoch": 0.13758190974273946,
+      "grad_norm": 0.18577855825424194,
+      "learning_rate": 0.00013732203389830509,
+      "loss": 1.709193801879883,
+      "step": 1950
+    },
+    {
+      "epoch": 0.13828745799783043,
+      "grad_norm": 0.16675390303134918,
+      "learning_rate": 0.00013698305084745765,
+      "loss": 1.749222755432129,
+      "step": 1960
+    },
+    {
+      "epoch": 0.1389930062529214,
+      "grad_norm": 0.1641705483198166,
+      "learning_rate": 0.00013664406779661018,
+      "loss": 1.6512624740600585,
+      "step": 1970
+    },
+    {
+      "epoch": 0.13969855450801238,
+      "grad_norm": 0.17839524149894714,
+      "learning_rate": 0.0001363050847457627,
+      "loss": 1.7599906921386719,
+      "step": 1980
+    },
+    {
+      "epoch": 0.14040410276310336,
+      "grad_norm": 0.17945389449596405,
+      "learning_rate": 0.00013596610169491527,
+      "loss": 1.7960474014282226,
+      "step": 1990
+    },
+    {
+      "epoch": 0.14110965101819434,
+      "grad_norm": 0.17191287875175476,
+      "learning_rate": 0.0001356271186440678,
+      "loss": 1.6708562850952149,
+      "step": 2000
+    },
+    {
+      "epoch": 0.1418151992732853,
+      "grad_norm": 0.175529345870018,
+      "learning_rate": 0.00013528813559322036,
+      "loss": 1.745585060119629,
+      "step": 2010
+    },
+    {
+      "epoch": 0.14252074752837626,
+      "grad_norm": 0.185764342546463,
+      "learning_rate": 0.00013494915254237287,
+      "loss": 1.7250106811523438,
+      "step": 2020
+    },
+    {
+      "epoch": 0.14322629578346724,
+      "grad_norm": 0.17936058342456818,
+      "learning_rate": 0.00013461016949152543,
+      "loss": 1.6920719146728516,
+      "step": 2030
+    },
+    {
+      "epoch": 0.14393184403855822,
+      "grad_norm": 0.17474661767482758,
+      "learning_rate": 0.000134271186440678,
+      "loss": 1.7728315353393556,
+      "step": 2040
+    },
+    {
+      "epoch": 0.1446373922936492,
+      "grad_norm": 0.17130956053733826,
+      "learning_rate": 0.00013393220338983052,
+      "loss": 1.646556282043457,
+      "step": 2050
+    },
+    {
+      "epoch": 0.14534294054874017,
+      "grad_norm": 0.17176896333694458,
+      "learning_rate": 0.00013359322033898305,
+      "loss": 1.7789287567138672,
+      "step": 2060
+    },
+    {
+      "epoch": 0.14604848880383112,
+      "grad_norm": 0.17843399941921234,
+      "learning_rate": 0.0001332542372881356,
+      "loss": 1.6941972732543946,
+      "step": 2070
+    },
+    {
+      "epoch": 0.1467540370589221,
+      "grad_norm": 0.17048513889312744,
+      "learning_rate": 0.00013291525423728815,
+      "loss": 1.729957389831543,
+      "step": 2080
+    },
+    {
+      "epoch": 0.14745958531401307,
+      "grad_norm": 0.16951288282871246,
+      "learning_rate": 0.00013257627118644068,
+      "loss": 1.6767967224121094,
+      "step": 2090
+    },
+    {
+      "epoch": 0.14816513356910405,
+      "grad_norm": 0.16641567647457123,
+      "learning_rate": 0.0001322372881355932,
+      "loss": 1.771189308166504,
+      "step": 2100
+    },
+    {
+      "epoch": 0.14887068182419502,
+      "grad_norm": 0.17894308269023895,
+      "learning_rate": 0.00013189830508474577,
+      "loss": 1.6634632110595704,
+      "step": 2110
+    },
+    {
+      "epoch": 0.149576230079286,
+      "grad_norm": 0.18185418844223022,
+      "learning_rate": 0.0001315593220338983,
+      "loss": 1.6945396423339845,
+      "step": 2120
+    },
+    {
+      "epoch": 0.15028177833437695,
+      "grad_norm": 0.18783807754516602,
+      "learning_rate": 0.00013122033898305087,
+      "loss": 1.7321378707885742,
+      "step": 2130
+    },
+    {
+      "epoch": 0.15098732658946792,
+      "grad_norm": 0.1784185916185379,
+      "learning_rate": 0.0001308813559322034,
+      "loss": 1.7040910720825195,
+      "step": 2140
+    },
+    {
+      "epoch": 0.1516928748445589,
+      "grad_norm": 0.18173034489154816,
+      "learning_rate": 0.00013054237288135593,
+      "loss": 1.722112274169922,
+      "step": 2150
+    },
+    {
+      "epoch": 0.15239842309964988,
+      "grad_norm": 0.16707691550254822,
+      "learning_rate": 0.0001302033898305085,
+      "loss": 1.7193023681640625,
+      "step": 2160
+    },
+    {
+      "epoch": 0.15310397135474085,
+      "grad_norm": 0.1860317885875702,
+      "learning_rate": 0.00012986440677966102,
+      "loss": 1.6834558486938476,
+      "step": 2170
+    },
+    {
+      "epoch": 0.1538095196098318,
+      "grad_norm": 0.1802806258201599,
+      "learning_rate": 0.00012952542372881356,
+      "loss": 1.7077167510986329,
+      "step": 2180
+    },
+    {
+      "epoch": 0.15451506786492278,
+      "grad_norm": 0.17852066457271576,
+      "learning_rate": 0.00012918644067796612,
+      "loss": 1.7088905334472657,
+      "step": 2190
+    },
+    {
+      "epoch": 0.15522061612001375,
+      "grad_norm": 0.17351694405078888,
+      "learning_rate": 0.00012884745762711865,
+      "loss": 1.6982431411743164,
+      "step": 2200
+    },
+    {
+      "epoch": 0.15592616437510473,
+      "grad_norm": 0.17839553952217102,
+      "learning_rate": 0.0001285084745762712,
+      "loss": 1.7794155120849608,
+      "step": 2210
+    },
+    {
+      "epoch": 0.1566317126301957,
+      "grad_norm": 0.18081575632095337,
+      "learning_rate": 0.00012816949152542374,
+      "loss": 1.661355209350586,
+      "step": 2220
+    },
+    {
+      "epoch": 0.15733726088528668,
+      "grad_norm": 0.17477205395698547,
+      "learning_rate": 0.00012783050847457627,
+      "loss": 1.7030864715576173,
+      "step": 2230
+    },
+    {
+      "epoch": 0.15804280914037763,
+      "grad_norm": 0.183249369263649,
+      "learning_rate": 0.0001274915254237288,
+      "loss": 1.7025772094726563,
+      "step": 2240
+    },
+    {
+      "epoch": 0.1587483573954686,
+      "grad_norm": 0.1709592640399933,
+      "learning_rate": 0.00012715254237288137,
+      "loss": 1.6538223266601562,
+      "step": 2250
+    },
+    {
+      "epoch": 0.15945390565055959,
+      "grad_norm": 0.17617081105709076,
+      "learning_rate": 0.00012681355932203393,
+      "loss": 1.6893890380859375,
+      "step": 2260
+    },
+    {
+      "epoch": 0.16015945390565056,
+      "grad_norm": 0.17261160910129547,
+      "learning_rate": 0.00012647457627118643,
+      "loss": 1.7037599563598633,
+      "step": 2270
+    },
+    {
+      "epoch": 0.16086500216074154,
+      "grad_norm": 0.17880302667617798,
+      "learning_rate": 0.000126135593220339,
+      "loss": 1.7065811157226562,
+      "step": 2280
+    },
+    {
+      "epoch": 0.16157055041583251,
+      "grad_norm": 0.17731013894081116,
+      "learning_rate": 0.00012579661016949152,
+      "loss": 1.6522327423095704,
+      "step": 2290
+    },
+    {
+      "epoch": 0.16227609867092346,
+      "grad_norm": 0.20257200300693512,
+      "learning_rate": 0.00012545762711864408,
+      "loss": 1.7568342208862304,
+      "step": 2300
+    },
+    {
+      "epoch": 0.16298164692601444,
+      "grad_norm": 0.17347396910190582,
+      "learning_rate": 0.00012511864406779662,
+      "loss": 1.7142282485961915,
+      "step": 2310
+    },
+    {
+      "epoch": 0.16368719518110542,
+      "grad_norm": 0.17510099709033966,
+      "learning_rate": 0.00012477966101694915,
+      "loss": 1.7098133087158203,
+      "step": 2320
+    },
+    {
+      "epoch": 0.1643927434361964,
+      "grad_norm": 0.17066830396652222,
+      "learning_rate": 0.0001244406779661017,
+      "loss": 1.6800745010375977,
+      "step": 2330
+    },
+    {
+      "epoch": 0.16509829169128737,
+      "grad_norm": 0.18264837563037872,
+      "learning_rate": 0.00012410169491525424,
+      "loss": 1.6962862014770508,
+      "step": 2340
+    },
+    {
+      "epoch": 0.16580383994637835,
+      "grad_norm": 0.1899186074733734,
+      "learning_rate": 0.00012376271186440678,
+      "loss": 1.7274456024169922,
+      "step": 2350
+    },
+    {
+      "epoch": 0.1665093882014693,
+      "grad_norm": 0.17581011354923248,
+      "learning_rate": 0.00012342372881355934,
+      "loss": 1.6959051132202148,
+      "step": 2360
+    },
+    {
+      "epoch": 0.16721493645656027,
+      "grad_norm": 0.2042885571718216,
+      "learning_rate": 0.00012308474576271187,
+      "loss": 1.656300926208496,
+      "step": 2370
+    },
+    {
+      "epoch": 0.16792048471165125,
+      "grad_norm": 0.17475232481956482,
+      "learning_rate": 0.00012274576271186443,
+      "loss": 1.656242561340332,
+      "step": 2380
+    },
+    {
+      "epoch": 0.16862603296674222,
+      "grad_norm": 0.18513426184654236,
+      "learning_rate": 0.00012240677966101693,
+      "loss": 1.7310556411743163,
+      "step": 2390
+    },
+    {
+      "epoch": 0.1693315812218332,
+      "grad_norm": 0.17900757491588593,
+      "learning_rate": 0.0001220677966101695,
+      "loss": 1.716302490234375,
+      "step": 2400
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 6000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 100,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.254671855999699e+17,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:327b53a5b2fd2ef8144fb3776c9c3db5c003cdf7c253e09cd5103f249e1d1aeb
+size 5713