|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- Norquinal/claude_multiround_chat_30k |
|
|
- ehartford/dolphin |
|
|
- BAAI/COIG-PC |
|
|
- Open-Orca/OpenOrca |
|
|
- vikp/textbook_quality_programming |
|
|
--- |
|
|
|
|
|
|
|
|
# RWKV v4 world 7B 65k context |
|
|
|
|
|
This is the model to replace the old rwkv 65k claude model, with special token and lower learning rate to maintain model former abilities. |
|
|
and trained a lots of English high quality textbooks and chinese novels with 65k context length. |
|
|
|
|
|
using it with rwkv runner only need 16G vram.(https://github.com/josStorer/RWKV-Runner) |
|
|
|
|
|
## contributor |
|
|
|
|
|
[@KevinMr](https://huggingface.co/KevinMr) |
|
|
[@Remixa](https://huggingface.co/Remixa) |
|
|
|
|
|
## trainning details |
|
|
https://wandb.ai/one-/one-rwkv-64k/runs/jn05hyc4 |
|
|
|
|
|
 |
|
|
|
|
|
|
|
|
|
|
|
## Testcase |
|
|
https://rwkv-next-web.ai-creator.net/ (temporary) |
|
|
|
|
|
https://rwkv.ai-creator.net/risu |
|
|
|
|
|
|
|
|
|
|
|
## how to use |
|
|
use vocabs files in runner config |
|
|
|
|
|
 |
|
|
|
|
|
|
|
|
|
|
|
|
