LSX-UniWue
/

ModernGBERT_1B

Feature Extraction

text-embeddings-inference

Model card Files Files and versions

Adjust max length in tokenizer config

#4

by aehrm - opened Jun 13

base: refs/heads/main

←

from: refs/pr/4

Discussion Files changed

Files changed (1) hide show

tokenizer_config.json +2 -2

tokenizer_config.json CHANGED Viewed

@@ -46,8 +46,8 @@
   "do_basic_tokenize": true,
   "do_lower_case": false,
   "mask_token": "[MASK]",
-  "max_len": 1024,
-  "model_max_length": 1024,
   "never_split": null,
   "pad_token": "[PAD]",
   "sep_token": "[SEP]",

   "do_basic_tokenize": true,
   "do_lower_case": false,
   "mask_token": "[MASK]",
+  "max_len": 8192,
+  "model_max_length": 8192,
   "never_split": null,
   "pad_token": "[PAD]",
   "sep_token": "[SEP]",