Commit
·
7202730
1
Parent(s):
b0ef4a0
Update README.md
Browse files
README.md
CHANGED
|
@@ -35,15 +35,6 @@ Code Llama is a collection of pretrained and fine-tuned generative text models r
|
|
| 35 |
|
| 36 |
**Model Architecture** Code Llama is an auto-regressive language model that uses an optimized transformer architecture.
|
| 37 |
|
| 38 |
-
|
| 39 |
-
||Training Data|Params|Content Length|GQA|Tokens|LR|
|
| 40 |
-
|---|---|---|---|---|---|---|
|
| 41 |
-
|Llama 2|*A new mix of publicly available online data*|7B|4k|✗|2.0T|3.0 x 10<sup>-4</sup>|
|
| 42 |
-
|Llama 2|*A new mix of publicly available online data*|13B|4k|✗|2.0T|3.0 x 10<sup>-4</sup>|
|
| 43 |
-
|Llama 2|*A new mix of publicly available online data*|70B|4k|✔|2.0T|1.5 x 10<sup>-4</sup>|
|
| 44 |
-
|
| 45 |
-
*Llama 2 family of models.* Token counts refer to pretraining data only. All models are trained with a global batch-size of 4M tokens. Bigger models - 70B -- use Grouped-Query Attention (GQA) for improved inference scalability.
|
| 46 |
-
|
| 47 |
**Model Dates** Code Llama and its variants have been trained between January 2023 and July 2023.
|
| 48 |
|
| 49 |
**Status** This is a static model trained on an offline dataset. Future versions of Code Llama - Instruct will be released as we improve model safety with community feedback.
|
|
|
|
| 35 |
|
| 36 |
**Model Architecture** Code Llama is an auto-regressive language model that uses an optimized transformer architecture.
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
**Model Dates** Code Llama and its variants have been trained between January 2023 and July 2023.
|
| 39 |
|
| 40 |
**Status** This is a static model trained on an offline dataset. Future versions of Code Llama - Instruct will be released as we improve model safety with community feedback.
|