OpenMOSS-Team
/

moss-moon-003-base

Text Generation

Model card Files Files and versions

xyliu commited on Apr 20, 2023

Commit

1478dc1

·

1 Parent(s): 896e51a

Update tokenization_moss.py

Files changed (1) hide show

tokenization_moss.py +1 -1

tokenization_moss.py CHANGED Viewed

@@ -136,7 +136,7 @@ class MossTokenizer(PreTrainedTokenizer):
         errors="replace",
         unk_token="<|endoftext|>",
         bos_token="<|endoftext|>",
-        eos_token="<eom>",
         pad_token=None,
         add_prefix_space=False,
         add_bos_token=False,

         errors="replace",
         unk_token="<|endoftext|>",
         bos_token="<|endoftext|>",
+        eos_token="<|endoftext|>",
         pad_token=None,
         add_prefix_space=False,
         add_bos_token=False,