newmindai
/

Llama-3.1-8B-Instruct-w16a16-4nodes-bs32

Text Generation

instruction-tuned

mixed-precision

question-answering

distributed-training

text-generation-inference

Model card Files Files and versions

nmmursit commited on 7 days ago

Commit

8636a51

·

verified ·

1 Parent(s): 5bd50d3

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -111,8 +111,8 @@ model = fully_shard(model, **fsdp_kwargs)
 | Llama-3.1-8B-Instruct-w16a8-4nodes-bs32  | 31478468 | 39.75          | 4     | 4    | **2.650** | **10.600** | 4           | 4          | 8                     | 512              |
 | Llama-3.1-8B-Instruct-w16a16-8nodes-bs32 | 31476914 | 22.00          | 8     | 4    | **2.933** | **11.733** | 4           | 4          | 8                     | 1024             |
 | Llama-3.1-8B-Instruct-w16a8-8nodes-bs32  | 31476844 | 23.50          | 8     | 4    | **3.133** | **12.533** | 4           | 4          | 8                     | 1024             |
-| Llama-3.1-8B-Instruct-w16a16-8nodes-bs64 | 31476914 | 22.00          | 8     | 4    | **2.933** | **11.733** | 4           | 4          | 8                     | 1024             |
-| Llama-3.1-8B-Instruct-w16a8-8nodes-bs64  | 31476844 | 23.50          | 8     | 4    | **3.133** | **12.533** | 4           | 4          | 8                     | 1024             |
 # *All 6-models trained on(1Node,4Noes,8Nodes with both bfp16-fp8 && bfp16 configurations)*
 | perplexity metric results for bfp16 && bfp16-fp8 configurations | Accuracy metric results for bfp16 && bfp16-fp8 configurations | Loss metric results for bfp16 && bfp16-fp8 configurations | Memory allocation for bfp16 && bfp16-fp8 configurations | Utilization for bfp16 && bfp16-fp8 configurations |

 | Llama-3.1-8B-Instruct-w16a8-4nodes-bs32  | 31478468 | 39.75          | 4     | 4    | **2.650** | **10.600** | 4           | 4          | 8                     | 512              |
 | Llama-3.1-8B-Instruct-w16a16-8nodes-bs32 | 31476914 | 22.00          | 8     | 4    | **2.933** | **11.733** | 4           | 4          | 8                     | 1024             |
 | Llama-3.1-8B-Instruct-w16a8-8nodes-bs32  | 31476844 | 23.50          | 8     | 4    | **3.133** | **12.533** | 4           | 4          | 8                     | 1024             |
+| Llama-3.1-8B-Instruct-w16a16-8nodes-bs64 | 31476914 | 22.00          | 8     | 4    | **2.933** | **11.733** | 4           | 8          | 8                     | 1024             |
+| Llama-3.1-8B-Instruct-w16a8-8nodes-bs64  | 31476844 | 23.50          | 8     | 4    | **3.133** | **12.533** | 4           | 8          | 8                     | 1024             |
 # *All 6-models trained on(1Node,4Noes,8Nodes with both bfp16-fp8 && bfp16 configurations)*
 | perplexity metric results for bfp16 && bfp16-fp8 configurations | Accuracy metric results for bfp16 && bfp16-fp8 configurations | Loss metric results for bfp16 && bfp16-fp8 configurations | Memory allocation for bfp16 && bfp16-fp8 configurations | Utilization for bfp16 && bfp16-fp8 configurations |