tokenizer-eval/tokenizer-data
Viewer
• Updated
• 24.8k • 23
Tokenizer Eval
non-profit
AI & ML interests
Tokenizers
Recent Activity
View all activity
christopher
updated 2 datasets 13 days ago
benjamin
authored a paper 3 months ago
benjamin
submitted a paper to Daily Papers 3 months ago
christopher
authored a paper 3 months ago
sanderland
authored a paper 4 months ago
christopher
authored a paper 5 months ago
christopher
posted an
update 5 months ago
sanderland
authored a paper 9 months ago
sanderland
authored 2 papers 11 months ago
benjamin
authored 2 papers 11 months ago
christopher
posted an
update over 1 year ago
Post
2111
The folks at Foursquare released a dataset of 104.5 million places of interest ( foursquare/fsq-os-places) and here's all of them on a plot
-
- 4 replies
christopher
posted an
update over 1 year ago
Post
2452
The Lichess database of games, puzzles, and engine evaluations is now on the Hub: 
Lichess
Billions of chess data points to download, query, and stream and we're excited to see what you'll build with it! ♟️ 🤗
- https://huggingface.co/collections/Lichess/positions-datasets-66f50837db5cd3287d60d489
- https://huggingface.co/collections/Lichess/games-datasets-66f508df78f4b43e1bb2d353
Billions of chess data points to download, query, and stream and we're excited to see what you'll build with it! ♟️ 🤗
- https://huggingface.co/collections/Lichess/positions-datasets-66f50837db5cd3287d60d489
- https://huggingface.co/collections/Lichess/games-datasets-66f508df78f4b43e1bb2d353
christopher
posted an
update over 1 year ago
Post
1346
4 million chess puzzles
benjamin
authored a paper over 1 year ago
benjamin
authored 3 papers almost 2 years ago
Zero-Shot Tokenizer Transfer
Paper
• 2405.07883 • Published
• 5
Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation
Paper
• 2305.18893 • Published
• 2
CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models
Paper
• 2305.14214 • Published