gigant
/

bytes-tokenizer

Model card Files Files and versions

gigant commited on Sep 18

Commit

9394d37

·

verified ·

1 Parent(s): f2caa95

Update README.md

Files changed (1) hide show

README.md +9 -3

README.md CHANGED Viewed

@@ -1,3 +1,9 @@
----
-license: mit
----

+---
+license: mit
+---
+```python
+from tokenizers import Tokenizer
+tokenizer = Tokenizer.from_file(tokenizer_path)
+```