Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,39 @@
+Grok 2 Community License Agreement
+Last Updated:  August 23, 2025
+
+1. Background and Definitions
+By downloading, accessing, or using the Materials (as defined below) relating to Grok 2 provided by X.AI LLC (“xAI”), you (“Licensee” or “you”) agree to the terms of this agreement (“Agreement”). If you accept this Agreement for or on behalf of an entity, you represent that you have the authority to bind that entity. As used in this Agreement, “Materials” means the Grok 2 materials provided to you by xAI under this Agreement, consisting of: (1) one or more machine learning models (including architecture and parameters); and (2) related artifacts (including associated data, documentation, and software) that are provided to you hereunder.  
+
+2. License Grant & Scope
+a. Permitted Uses: xAI grants you a non-exclusive, worldwide, revocable license to use, reproduce, distribute, and modify the Materials:
+	•	For non-commercial and research purposes; and
+	•	For commercial use solely if you and your affiliates abide by all of the guardrails provided in xAI's Acceptable Use Policy (https://x.ai/legal/acceptable-use-policy), including 1. Comply with the law, 2. Do not harm people or property, and 3. Respect guardrails and don't mislead.
+b. Restrictions:
+	•	You may not use the Materials, derivatives, or outputs (including generated data) to train, create, or improve any foundational, large language, or general-purpose AI models, except for modifications or fine-tuning of Grok 2 permitted under and in accordance with the terms of this Agreement.
+	•	No right to use xAI’s trademarks is granted, except as required for attribution (see below).
+
+3. Distribution & Attribution
+If you distribute the Materials, derivatives, or products/services incorporating them:
+	•	Include this Agreement and a notice stating: “This product includes materials licensed under the xAI Community License. Copyright © 2025 xAI. All rights reserved.”
+	•	Prominently display “Powered by xAI” in related materials or interfaces.
+
+4. Ownership & Outputs
+xAI retains all rights in the Materials. This Agreement does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs generated by using the Materials. If you provide feedback, suggestions, or ideas, you grant xAI a perpetual, worldwide, irrevocable, royalty-free license to use and incorporate that feedback without restriction.
+
+5. Acceptable Use
+You are responsible for implementing appropriate safety measures, including filters and human oversight, suitable for your use case. You must comply with xAI’s Acceptable Use Policy (AUP), as well as all applicable laws. You may not use the Materials for illegal, harmful, or abusive activities.
+
+6. Warranty Disclaimer & Limitation of Liability
+THE MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NONINFRINGEMENT, ACCURACY, OR THE ABSENCE OF LATENT OR OTHER DEFECTS OR ERRORS, WHETHER OR NOT DISCOVERABLE, ALL TO THE GREATEST EXTENT PERMISSIBLE UNDER APPLICABLE LAW.
+YOU ARE SOLELY RESPONSIBLE FOR (1) CLEARING RIGHTS OF OTHER PERSONS THAT MAY APPLY TO THE MATERIALS OR ANY USE THEREOF, INCLUDING WITHOUT LIMITATION ANY PERSON'S COPYRIGHTS OR OTHER RIGHTS INCLUDED OR EMBODIED IN THE MATERIALS; (2) OBTAINING ANY NECESSARY CONSENTS, PERMISSIONS OR OTHER RIGHTS REQUIRED FOR ANY USE OF THE MATERIALS; OR (3) PERFORMING ANY DUE DILIGENCE OR UNDERTAKING ANY OTHER INVESTIGATIONS INTO THE MATERIALS OR ANYTHING INCORPORATED OR EMBODIED THEREIN.
+IN NO EVENT SHALL XAI BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE MATERIALS, THE USE THEREOF, OR OTHER DEALINGS THEREIN. TO THE MAXIMUM EXTENT PERMITTED BY LAW, XAI WILL NOT BE LIABLE FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, OR FOR AGGREGATE LIABILITY EXCEEDING $100, REGARDLESS OF THE LEGAL THEORY.
+
+7. Termination
+This license terminates immediately upon your breach or if you exceed the permitted commercial threshold. Upon termination, you must cease all use and delete all copies of the Materials and derivatives.
+Additionally, if you file, maintain, or voluntarily participate in a lawsuit against any person or entity alleging that the Materials, or any part thereof, directly or indirectly infringe any patent, then your license under this Agreement shall immediately terminate. This does not apply to a lawsuit brought in response to a corresponding lawsuit first filed against you.
+
+8. Governing Law
+The laws of Texas govern this Agreement, and any dispute shall be resolved exclusively in the courts located in Tarrant County, Texas.
+
+9. Miscellaneous
+This Agreement is the entire agreement between the parties on this subject. Failure to enforce any provision is not a waiver. If any provision is unenforceable, the remainder remains in effect. xAI may assign this Agreement, including in connection with a merger or acquisition. Licensee may not assign this Agreement without xAI’s prior written consent. This Agreement creates no third-party beneficiaries. You must comply with all applicable export control, trade compliance, and sanctions laws.

README.md

@@ -0,0 +1,68 @@
+---
+library_name: transformers
+tags:
+- tokenizers
+- sglang
+license: other
+license_name: grok-2
+license_link: https://huggingface.co/xai-org/grok-2/blob/main/LICENSE
+---
+
+# Grok-2 Tokenizer
+
+A 🤗-compatible version of the **Grok-2 tokenizer** (adapted from [xai-org/grok-2](https://huggingface.co/xai-org/grok-2)).
+
+This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers),
+[Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
+
+## Motivation
+
+As Grok 2.5 aka. [xai-org/grok-2](https://github.com/xai-org/grok-2) has been recently released on the 🤗 Hub with [SGLang](https://github.com/sgl-project/sglang)
+native support, but the checkpoints on the Hub won't come with a Hugging Face compatible tokenizer, but rather with a `tiktoken`-based
+JSON export, which is [internally read and patched in SGLang](https://github.com/sgl-project/sglang/blob/fd71b11b1d96d385b09cb79c91a36f1f01293639/python/sglang/srt/tokenizer/tiktoken_tokenizer.py#L29-L108).
+
+This repository then contains the Hugging Face compatible export so that users can easily interact and play around with the Grok-2 tokenizer,
+besides that allowing to use it via SGLang without having to pull the repository manually from the Hub and then using a mount, to prevent from directly having
+to point to the tokenizer path, so that Grok-2 can be deployed as:
+
+```bash
+python3 -m sglang.launch_server --model-path xai-org/grok-2 --tokenizer-path alvarobartt/grok-2-tokenizer --tp-size 8 --quantization fp8 --attention-backend triton
+```
+
+Rather than the former 2-step process:
+
+```bash
+hf download xai-org/grok-2 --local-dir /local/grok-2
+
+python3 -m sglang.launch_server --model-path /local/grok-2 --tokenizer-path /local/grok-2/tokenizer.tok.json --tp-size 8 --quantization fp8 --attention-backend triton
+```
+
+## Example
+
+```py
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained("alvarobartt/grok-2-tokenizer")
+
+assert tokenizer.encode("Human: What is Deep Learning?<|separator|>\n\n") == [
+    35406,
+    186,
+    2171,
+    458,
+    17454,
+    14803,
+    191,
+    1,
+    417,
+]
+
+assert (
+    tokenizer.apply_chat_template(
+        [{"role": "user", "content": "What is the capital of France?"}], tokenize=False
+    )
+    == "Human: What is the capital of France?<|separator|>\n\n"
+)
+```
+
+> [!NOTE]
+> This repository has been inspired by earlier similar work by [Xenova](https://huggingface.co/Xenova) in [`Xenova/grok-1-tokenizer`](https://huggingface.co/Xenova/grok-1-tokenizer).

chat_template.jinja

@@ -0,0 +1,7 @@
+{% for message in messages %}{% if message['role'] == 'user' %}{{ 'Human: ' + message['content'].strip() + '<|separator|>
+
+' }}{% elif message['role'] == 'system' %}{{ 'System: ' + message['content'].strip() + '<|separator|>
+
+' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: '  + message['content'] + '<|separator|>
+
+' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}

config.json

@@ -0,0 +1,36 @@
+{
+  "architectures": [
+    "Grok1ForCausalLM"
+  ],
+  "embedding_multiplier_scale": 90.50966799187809,
+  "output_multiplier_scale": 0.5,
+  "vocab_size": 131072,
+  "hidden_size": 8192,
+  "intermediate_size": 32768,
+  "moe_intermediate_size": 16384,
+  "max_position_embeddings": 131072,
+  "num_experts_per_tok": 2,
+  "num_local_experts": 8,
+  "residual_moe": true,
+  "num_attention_heads": 64,
+  "num_key_value_heads": 8,
+  "num_hidden_layers": 64,
+  "head_dim": 128,
+  "rms_norm_eps": 1e-05,
+  "final_logit_softcapping": 50,
+  "attn_logit_softcapping": 30.0,
+  "router_logit_softcapping": 30.0,
+  "rope_theta": 208533496,
+  "attn_temperature_len": 1024,
+  "sliding_window_size": -1,
+  "global_attn_every_n": 1,
+  "model_type": "git",
+  "torch_dtype": "bfloat16",
+  "rope_type": "original",
+  "original_max_position_embeddings": 8192,
+  "scaling_factor": 16.0,
+  "extrapolation_factor": 1.0,
+  "attn_factor": 1.0,
+  "beta_fast": 8,
+  "beta_slow": 1
+}

pytorch_model-00000-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:315d2eb626636477b35a19c60b481ae43b70fc7b9d24645255ded00cecf2a6bb
+size 2147483760

pytorch_model-00001-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f8001b2d7baaee0bd5bce3ad41d117148c591e485dc8a838b8abccb650ef9bb
+size 2147483744

pytorch_model-00002-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e41eeadefc5da657bac925e752dab11672d7ccee729ed60df8c32df60c6fcd6e
+size 16472

pytorch_model-00003-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5117eeb28c1447403df924fa93c62a9ea09d2d6fa773e27367fc5e6687b10b99
+size 34359745872

pytorch_model-00004-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6921cc3380cfbd783a3b52bccf81bc5c7c645548f9f2f1b86279649edbba3cf0
+size 34359745872

pytorch_model-00005-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90bb1c9d09d31b58becb2cf13f1694166708a52cd97315185c6a7ddbcc64f1ca
+size 34359745744

pytorch_model-00006-TP-000.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5128445f49d89ed53463d6e454ef9f466ca65e2607a006761a558f9fd60754d
+size 17179936544

pytorch_model-00006-TP-001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9f0869351c7a2baac725b2b2903dec79b9afb0f49f2004ce3a1f35763708e40
+size 17179936544

pytorch_model-00006-TP-002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:930f397a72ed323783e3f029bc1252beb9f2d2bce69610fa0f094e361547d1dc
+size 17179936544

pytorch_model-00006-TP-003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b266d9293692e69cc2decbf1914916cde492de8b485e7ad670447674e5170e49
+size 17179936544

pytorch_model-00006-TP-004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be0f2daffdc5a7c031ce685cc5bd89fe9058eedb14a107855ea45e272b7022c0
+size 17179936544

pytorch_model-00006-TP-005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4697b87056edc930ec836528985fb25a40040417226e47a9c483786b81b56a7d
+size 17179936544

pytorch_model-00006-TP-006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d572dc1910937e16be0017118daca90734af892ea019581584c738abd74975d
+size 17179936544

pytorch_model-00006-TP-007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cddca91357cc76894e1704d0bba02dcddeb7e063329f30198a26f3eec98f1ca1
+size 17179936544

pytorch_model-00007-TP-000.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ca76131b44751d11ba6abafa0567939812c83af63ff3ed91771e5a03b28e395
+size 17179936544

pytorch_model-00007-TP-001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6689603a91fec740d69a67512a7d6c3d4176a2068fcc88e08a79646c7bc9a4ec
+size 17179936544

pytorch_model-00007-TP-002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91c5af3bc69e126adf09cf066b5554fd55a804cb246e48c769725044bf6c2482
+size 17179936544

pytorch_model-00007-TP-003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1643a8a743dd135e3aa082cb732ade54c624e188e264ce2ca29818eca4579129
+size 17179936544

pytorch_model-00007-TP-004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cce342c135799ea961f7c7335e1bd485190e7ed4511f1a4b25792af856b702b
+size 17179936544

pytorch_model-00007-TP-005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ecb1e412e72be6acf5a94300d20b922440710dbc3ff655081e14fde52dead193
+size 17179936544

pytorch_model-00007-TP-006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff8c1dc7ba5222e7e88eeaa8c71f9bcc6c72abb2642cd53708e89c3bb29e9a5e
+size 17179936544

pytorch_model-00007-TP-007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d890e53bae6eb64606791a70ec4055a0909eb75d704bc505050f87a8f4304776
+size 17179936544

pytorch_model-00008-TP-000.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6023ef14f89644bba6dea2b3cd268ef90a603a724e58e6f25fa4b35e09da805
+size 17179936544

pytorch_model-00008-TP-001.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:adf26b682c3898af5f1ab6fe106579d29aea00ab661e8d164988b6e8093f6412
+size 17179936544

pytorch_model-00008-TP-002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8381e5ff848f33a2a715155aec5303309d1044bb105d0267bbbbd702e2ea4181
+size 17179936544

pytorch_model-00008-TP-003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38471f261ce898dc7d1266fbabbacd922248767d4e909a0f431d2adaf77c9d1f
+size 17179936544

pytorch_model-00008-TP-004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22d33701fa88dff2ecadb388238a12c7a5d63c6de2dae120ae98ffacca109683
+size 17179936544

pytorch_model-00008-TP-005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abdaa52c585fe182cd82918bde1ec3fa2b12033a1c9c11cebd753a2b3db88355
+size 17179936544

pytorch_model-00008-TP-006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f50b318ba532e0032bd6f25b59b54852029e6a026ef19852baa7851b6d5d7004
+size 17179936544

pytorch_model-00008-TP-007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7846ffc97ad222ca88bb38771a97409e8a80b62ce3573c096378a76f97c54578
+size 17179936544

pytorch_model-00009-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cebb4ddcdb6520cb50dabdb0a5568e83de081dbebb00208e2a44afd82fd3c352
+size 1073749240

pytorch_model-00010-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db9376f06a12092da29d028c2ebfe1206b1c85c50513d5138e7442f02ede635c
+size 8589942120

pytorch_model-00011-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1306ca738a956652a3d6bea2df5f3899d0c42dd96f22d75c62f72792f236d7a7
+size 8589942120

pytorch_model-00012-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88ca01f2876be5a7b268f0cd068a8e49bc9daae48426322bb420eacfcf308703
+size 1073749240

pytorch_model-00013-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:924822ee882d55ac56938342ca435bcd2c4e71a7116dba90e6301845ab19f509
+size 1055096

pytorch_model-00014-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:658fddf97d85edcd527861c7785d9d4a7801d09799819f9de85aa7b2d29ecf9e
+size 1055160

pytorch_model-00015-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc94f4dff86cad956d7eae017ff7219d55e2d902d45378fb5502729020376c30
+size 1055032

pytorch_model-00016-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63b443a0a63169c76e6342304583351a50a571ce45070f32650ec0da0f8de08b
+size 1055096

pytorch_model-00017-TP-common.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49021a8d7c657eedb8bb83de222fe9d8015cb1db2388ee2e02ad017b1d289f32
+size 8395888

special_tokens_map.json

@@ -0,0 +1,135 @@
+{
+  "additional_special_tokens": [
+    "<|pad|>",
+    "<|separator|>",
+    "<|eos|>",
+    "<|control1|>",
+    "<|control2|>",
+    "<|control3|>",
+    "<|control4|>",
+    "<|control5|>",
+    "<|control6|>",
+    "<|control7|>",
+    "<|control8|>",
+    "<|control9|>",
+    "<|control10|>",
+    "<|control11|>",
+    "<|control12|>",
+    "<|control13|>",
+    "<|control14|>",
+    "<|control15|>",
+    "<|control16|>",
+    "<|control17|>",
+    "<|control18|>",
+    "<|control19|>",
+    "<|control20|>",
+    "<|control21|>",
+    "<|control22|>",
+    "<|control23|>",
+    "<|control24|>",
+    "<|control25|>",
+    "<|control26|>",
+    "<|control27|>",
+    "<|control28|>",
+    "<|control29|>",
+    "<|control30|>",
+    "<|control31|>",
+    "<|control32|>",
+    "<|control33|>",
+    "<|control34|>",
+    "<|control35|>",
+    "<|control36|>",
+    "<|control37|>",
+    "<|control38|>",
+    "<|control39|>",
+    "<|control40|>",
+    "<|control41|>",
+    "<|control42|>",
+    "<|control43|>",
+    "<|control44|>",
+    "<|control45|>",
+    "<|control46|>",
+    "<|control47|>",
+    "<|control48|>",
+    "<|control49|>",
+    "<|control50|>",
+    "<|control51|>",
+    "<|control52|>",
+    "<|control53|>",
+    "<|control54|>",
+    "<|control55|>",
+    "<|control56|>",
+    "<|control57|>",
+    "<|control58|>",
+    "<|control59|>",
+    "<|control60|>",
+    "<|control61|>",
+    "<|control62|>",
+    "<|control63|>",
+    "<|control64|>",
+    "<|control65|>",
+    "<|control66|>",
+    "<|control67|>",
+    "<|control68|>",
+    "<|control69|>",
+    "<|control70|>",
+    "<|control71|>",
+    "<|control72|>",
+    "<|control73|>",
+    "<|control74|>",
+    "<|control75|>",
+    "<|control76|>",
+    "<|control77|>",
+    "<|control78|>",
+    "<|control79|>",
+    "<|control80|>",
+    "<|control81|>",
+    "<|control82|>",
+    "<|control83|>",
+    "<|control84|>",
+    "<|control85|>",
+    "<|control86|>",
+    "<|control87|>",
+    "<|control88|>",
+    "<|control89|>",
+    "<|control90|>",
+    "<|control91|>",
+    "<|control92|>",
+    "<|control93|>",
+    "<|control94|>",
+    "<|control95|>",
+    "<|control96|>",
+    "<|control97|>",
+    "<|control98|>",
+    "<|control99|>",
+    "<|control100|>",
+    "<|control101|>",
+    "<|control102|>",
+    "<|control103|>",
+    "<|control104|>",
+    "<|control105|>",
+    "<|control106|>",
+    "<|control107|>",
+    "<|control108|>",
+    "<|control109|>",
+    "<|control110|>",
+    "<|control111|>",
+    "<|control112|>",
+    "<|control113|>",
+    "<|control114|>",
+    "<|control115|>",
+    "<|control116|>",
+    "<|control117|>",
+    "<|control118|>",
+    "<|control119|>",
+    "<|control120|>",
+    "<|control121|>",
+    "<|control122|>",
+    "<|control123|>",
+    "<|control124|>",
+    "<|control125|>"
+  ],
+  "eos_token": "<|eos|>",
+  "pad_token": "<|pad|>",
+  "sep_token": "<|separator|>"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63e538a53de71acf0585603a7f1f62cc66c5e6fa5763d753d10ad071f5c5d4fe
+size 17963143

tokenizer_config.json

@@ -0,0 +1,1165 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|separator|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|eos|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|control1|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|control2|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|control3|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<|control4|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<|control5|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<|control6|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<|control7|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<|control8|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<|control9|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<|control10|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<|control11|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<|control12|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<|control13|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<|control14|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "17": {
+      "content": "<|control15|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "18": {
+      "content": "<|control16|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "19": {
+      "content": "<|control17|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "20": {
+      "content": "<|control18|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "21": {
+      "content": "<|control19|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "22": {
+      "content": "<|control20|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "23": {
+      "content": "<|control21|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "24": {
+      "content": "<|control22|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25": {
+      "content": "<|control23|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "26": {
+      "content": "<|control24|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "27": {
+      "content": "<|control25|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "28": {
+      "content": "<|control26|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "29": {
+      "content": "<|control27|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30": {
+      "content": "<|control28|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "31": {
+      "content": "<|control29|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "32": {
+      "content": "<|control30|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "33": {
+      "content": "<|control31|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "34": {
+      "content": "<|control32|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "35": {
+      "content": "<|control33|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "36": {
+      "content": "<|control34|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "37": {
+      "content": "<|control35|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "38": {
+      "content": "<|control36|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "39": {
+      "content": "<|control37|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "40": {
+      "content": "<|control38|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "41": {
+      "content": "<|control39|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "42": {
+      "content": "<|control40|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "43": {
+      "content": "<|control41|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "44": {
+      "content": "<|control42|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "45": {
+      "content": "<|control43|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "46": {
+      "content": "<|control44|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "47": {
+      "content": "<|control45|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "48": {
+      "content": "<|control46|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49": {
+      "content": "<|control47|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50": {
+      "content": "<|control48|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "51": {
+      "content": "<|control49|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "52": {
+      "content": "<|control50|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "53": {
+      "content": "<|control51|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "54": {
+      "content": "<|control52|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "55": {
+      "content": "<|control53|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "56": {
+      "content": "<|control54|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "57": {
+      "content": "<|control55|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "58": {
+      "content": "<|control56|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "59": {
+      "content": "<|control57|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "60": {
+      "content": "<|control58|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "61": {
+      "content": "<|control59|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "62": {
+      "content": "<|control60|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "63": {
+      "content": "<|control61|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64": {
+      "content": "<|control62|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "65": {
+      "content": "<|control63|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "66": {
+      "content": "<|control64|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "67": {
+      "content": "<|control65|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "68": {
+      "content": "<|control66|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "69": {
+      "content": "<|control67|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "70": {
+      "content": "<|control68|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "71": {
+      "content": "<|control69|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "72": {
+      "content": "<|control70|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "73": {
+      "content": "<|control71|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "74": {
+      "content": "<|control72|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "75": {
+      "content": "<|control73|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "76": {
+      "content": "<|control74|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "77": {
+      "content": "<|control75|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "78": {
+      "content": "<|control76|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "79": {
+      "content": "<|control77|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "80": {
+      "content": "<|control78|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "81": {
+      "content": "<|control79|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "82": {
+      "content": "<|control80|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "83": {
+      "content": "<|control81|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "84": {
+      "content": "<|control82|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "85": {
+      "content": "<|control83|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "86": {
+      "content": "<|control84|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "87": {
+      "content": "<|control85|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "88": {
+      "content": "<|control86|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "89": {
+      "content": "<|control87|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "90": {
+      "content": "<|control88|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "91": {
+      "content": "<|control89|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "92": {
+      "content": "<|control90|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "93": {
+      "content": "<|control91|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "94": {
+      "content": "<|control92|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "95": {
+      "content": "<|control93|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "96": {
+      "content": "<|control94|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "97": {
+      "content": "<|control95|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "98": {
+      "content": "<|control96|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "99": {
+      "content": "<|control97|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "<|control98|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "<|control99|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "<|control100|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "<|control101|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "104": {
+      "content": "<|control102|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "105": {
+      "content": "<|control103|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "106": {
+      "content": "<|control104|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "107": {
+      "content": "<|control105|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "108": {
+      "content": "<|control106|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "109": {
+      "content": "<|control107|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "110": {
+      "content": "<|control108|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "111": {
+      "content": "<|control109|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "112": {
+      "content": "<|control110|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "113": {
+      "content": "<|control111|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "114": {
+      "content": "<|control112|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "115": {
+      "content": "<|control113|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "116": {
+      "content": "<|control114|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "117": {
+      "content": "<|control115|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "118": {
+      "content": "<|control116|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "119": {
+      "content": "<|control117|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "120": {
+      "content": "<|control118|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "121": {
+      "content": "<|control119|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "122": {
+      "content": "<|control120|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "123": {
+      "content": "<|control121|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "124": {
+      "content": "<|control122|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "125": {
+      "content": "<|control123|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "126": {
+      "content": "<|control124|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "127": {
+      "content": "<|control125|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|pad|>",
+    "<|separator|>",
+    "<|eos|>",
+    "<|control1|>",
+    "<|control2|>",
+    "<|control3|>",
+    "<|control4|>",
+    "<|control5|>",
+    "<|control6|>",
+    "<|control7|>",
+    "<|control8|>",
+    "<|control9|>",
+    "<|control10|>",
+    "<|control11|>",
+    "<|control12|>",
+    "<|control13|>",
+    "<|control14|>",
+    "<|control15|>",
+    "<|control16|>",
+    "<|control17|>",
+    "<|control18|>",
+    "<|control19|>",
+    "<|control20|>",
+    "<|control21|>",
+    "<|control22|>",
+    "<|control23|>",
+    "<|control24|>",
+    "<|control25|>",
+    "<|control26|>",
+    "<|control27|>",
+    "<|control28|>",
+    "<|control29|>",
+    "<|control30|>",
+    "<|control31|>",
+    "<|control32|>",
+    "<|control33|>",
+    "<|control34|>",
+    "<|control35|>",
+    "<|control36|>",
+    "<|control37|>",
+    "<|control38|>",
+    "<|control39|>",
+    "<|control40|>",
+    "<|control41|>",
+    "<|control42|>",
+    "<|control43|>",
+    "<|control44|>",
+    "<|control45|>",
+    "<|control46|>",
+    "<|control47|>",
+    "<|control48|>",
+    "<|control49|>",
+    "<|control50|>",
+    "<|control51|>",
+    "<|control52|>",
+    "<|control53|>",
+    "<|control54|>",
+    "<|control55|>",
+    "<|control56|>",
+    "<|control57|>",
+    "<|control58|>",
+    "<|control59|>",
+    "<|control60|>",
+    "<|control61|>",
+    "<|control62|>",
+    "<|control63|>",
+    "<|control64|>",
+    "<|control65|>",
+    "<|control66|>",
+    "<|control67|>",
+    "<|control68|>",
+    "<|control69|>",
+    "<|control70|>",
+    "<|control71|>",
+    "<|control72|>",
+    "<|control73|>",
+    "<|control74|>",
+    "<|control75|>",
+    "<|control76|>",
+    "<|control77|>",
+    "<|control78|>",
+    "<|control79|>",
+    "<|control80|>",
+    "<|control81|>",
+    "<|control82|>",
+    "<|control83|>",
+    "<|control84|>",
+    "<|control85|>",
+    "<|control86|>",
+    "<|control87|>",
+    "<|control88|>",
+    "<|control89|>",
+    "<|control90|>",
+    "<|control91|>",
+    "<|control92|>",
+    "<|control93|>",
+    "<|control94|>",
+    "<|control95|>",
+    "<|control96|>",
+    "<|control97|>",
+    "<|control98|>",
+    "<|control99|>",
+    "<|control100|>",
+    "<|control101|>",
+    "<|control102|>",
+    "<|control103|>",
+    "<|control104|>",
+    "<|control105|>",
+    "<|control106|>",
+    "<|control107|>",
+    "<|control108|>",
+    "<|control109|>",
+    "<|control110|>",
+    "<|control111|>",
+    "<|control112|>",
+    "<|control113|>",
+    "<|control114|>",
+    "<|control115|>",
+    "<|control116|>",
+    "<|control117|>",
+    "<|control118|>",
+    "<|control119|>",
+    "<|control120|>",
+    "<|control121|>",
+    "<|control122|>",
+    "<|control123|>",
+    "<|control124|>",
+    "<|control125|>"
+  ],
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|eos|>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "sep_token": "<|separator|>",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}