metadata
library_name: transformers
tags:
- transformers.js
- tokenizers
GPT-3.5-turbo Tokenizer
A 🤗-compatible version of the GPT-3.5-turbo tokenizer (adapted from openai/tiktoken). This means it can be used with Hugging Face libraries including Transformers, Tokenizers, and Transformers.js.
Usage (Transformers.js)
If you haven't already, you can install the Transformers.js JavaScript library from NPM using:
npm i @huggingface/transformers
Example: Tokenize text using Transformers.js
import { AutoTokenizer } from '@huggingface/transformers';
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/gpt-3.5-turbo');
const tokens = tokenizer.encode('hello world'); // [15339, 1917]
Example usage:
Transformers/Tokenizers
from transformers import GPT2TokenizerFast
tokenizer = GPT2TokenizerFast.from_pretrained('Xenova/gpt-3.5-turbo')
assert tokenizer.encode('hello world') == [15339, 1917]
Transformers.js
import { AutoTokenizer } from '@huggingface/transformers';
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/gpt-3.5-turbo');
const tokens = tokenizer.encode('hello world'); // [15339, 1917]