Update README.md
Browse files
README.md
CHANGED
|
@@ -12,7 +12,7 @@ language:
|
|
| 12 |
|
| 13 |
# ipt-350m
|
| 14 |
|
| 15 |
-
ipt-350m is a decoder-style transformer pretrained from scratch on ~13B tokens of Italian text.
|
| 16 |
|
| 17 |
It uses a modified transformer architecture optimized for efficient training and inference. Positional embeddings are replaced with Attention with Linear Biases ([ALiBi](https://arxiv.org/abs/2108.12409)).
|
| 18 |
|
|
|
|
| 12 |
|
| 13 |
# ipt-350m
|
| 14 |
|
| 15 |
+
ipt-350m is a decoder-style transformer pretrained from scratch on ~13B tokens of Italian text (wip: trained on unfiltered oscar).
|
| 16 |
|
| 17 |
It uses a modified transformer architecture optimized for efficient training and inference. Positional embeddings are replaced with Attention with Linear Biases ([ALiBi](https://arxiv.org/abs/2108.12409)).
|
| 18 |
|