Spaces:

---

smol-course

/

leaderboard

Like 14

Running

Submit onethreedlee/smollm3-unit3-dpo for Unit 3 (DPO).

Base model: HuggingFaceTB/SmolLM3-3B; preference data: Anthropic/hh-rlhf.

Full-parameter DPO ran for 1,000 optimizer steps with learning rate 5e-7, beta 0.1, seed 42, BF16, batch 1 × accumulation 8, max prompt length 512, and max sequence length 1,024. Of 16,000 sampled candidate pairs, 13,086 valid pairs fit the limits. The final held-out preference check used 54 pairs, yielding reward accuracy 0.4444; this does not establish alignment improvement.

Training job: https://huggingface.co/jobs/onethreedlee/6ac69b21df2184ac91ac3e5a

Training launch:

hf jobs uv run --python 3.12 --flavor a100-large --timeout 4h \
  --detach --secrets HF_TOKEN unit3-dpo/train.py \
  --hub-model-id onethreedlee/smollm3-unit3-dpo \
  --max-steps 1000 --train-samples 16000 --learning-rate 5e-7 --beta 0.1

The exact training script is preserved at https://huggingface.co/onethreedlee/smollm3-unit3-dpo/blob/db1d09716a2727a69a1d7a6be21d1ce2b2a1cceb/reproduce/train.py. It pins TRL 0.22.2, Transformers 4.56.2, and PyTorch 2.8.0, and pushes checkpoints plus final model/tokenizer to the Hub.

Evaluation job: https://huggingface.co/jobs/onethreedlee/6acc2162095c578089325c47

Evaluation launch:

hf jobs uv run --detach --name unit3-dpo-leaderboard-eval-v3 \
  --python 3.12 --flavor a10g-large --timeout 90m --secrets HF_TOKEN \
  unit3-dpo/evaluate.py --model onethreedlee/smollm3-unit3-dpo \
  --results-org onethreedlee --include-baseline

The evaluation script is preserved at https://huggingface.co/onethreedlee/smollm3-unit3-dpo/blob/db1d09716a2727a69a1d7a6be21d1ce2b2a1cceb/reproduce/evaluate.py. It pins Lighteval 0.10.0, vLLM 0.10.0, PyTorch 2.7.1, Transformers 4.56.2, datasets 4.8.5, huggingface_hub 0.36.2, NumPy 1.26.4, Arrow 21.0.0, SciPy 1.15.3, and xxhash <4.

Tasks: leaderboard|truthfulqa:mc|0|0,leaderboard|hellaswag|0|0,leaderboard|arc:challenge|0|0,lighteval|gsm8k|0|0. Full evaluation splits, zero shot, no sample limit. Both original and DPO models use their chat templates, the /no_think system instruction, BF16, greedy generation capped at 256 new tokens, and a 4,096-token context. Revisions and exact underlying Lighteval commands are recorded in the evaluation manifest.

GSM8K is included because the current Space loader reads lighteval|gsm8k|0 / extractive_match; the other three tasks follow the Unit 3 submission guide. These benchmarks do not establish safety.

Full DPO results

Public results dataset: https://huggingface.co/datasets/onethreedlee/details_onethreedlee__smollm3-unit3-dpo

All 13,350 questions were evaluated. No input prompts were truncated. The public dataset was loaded with load_dataset(..., "results"), and its latest split contains the exact lighteval|gsm8k|0 / extractive_match fields consumed by the current Space.

The original-model comparison is continuing as the second stage of the same job and is not yet available. This submission reports the completed DPO evaluation only; it does not claim improved alignment or safety.