kz919/llama3_1b_cautious_chinchilla_8142025

Model Details

This model is a 1B llama3 model pretrained from scratch with torchtitan on fineweb-edu with C_AdamW optimizer. 20x chinchilla rule for 20B tokens seen.

How to use

import torch
from transformers import pipeline


pipe = pipeline(
    "text-generation",
    model="kz919/llama3_1b_cautious_chinchilla_8132025",
)

print(pipe("The key to life is"))

Downstream Eval

ARC, Hellaswag, Lambda_OpenAI, OpenbookQA, PIQA

lm_eval --model hf --model_args pretrained=kz919/llama3_1b_cautious_chinchilla_8142025,dtype="bfloat16",add_bos_token=True --tasks lambada_openai,hellaswag,piqa,arc_easy,arc_challenge,openbookqa --device cuda:7 --batch_size 8

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_challenge	1	none	0	acc	↑	0.2730	±	0.0130
		none	0	acc_norm	↑	0.2765	±	0.0131
arc_easy	1	none	0	acc	↑	0.5960	±	0.0101
		none	0	acc_norm	↑	0.5290	±	0.0102
hellaswag	1	none	0	acc	↑	0.3442	±	0.0047
		none	0	acc_norm	↑	0.4122	±	0.0049
lambada_openai	1	none	0	acc	↑	0.3264	±	0.0065
		none	0	perplexity	↓	39.7510	±	1.6063
openbookqa	1	none	0	acc	↑	0.2200	±	0.0185
		none	0	acc_norm	↑	0.3300	±	0.0210
piqa	1	none	0	acc	↑	0.6872	±	0.0108
		none	0	acc_norm	↑	0.6850	±	0.0108

MMLU

Groups	Version	Filter	n-shot	Metric		Value		Stderr
mmlu	2	none		acc	↑	0.2536	±	0.0037
- humanities	2	none		acc	↑	0.2667	±	0.0064
- other	2	none		acc	↑	0.2475	±	0.0077
- social sciences	2	none		acc	↑	0.2337	±	0.0076
- stem	2	none		acc	↑	0.2594	±	0.0078