Article
Decoding Strategies in Large Language Models
•
101
Nishith Jain
KingNish
AI & ML interests
AI is fun actually.
Recent Activity
upvoted
an
article
3 days ago
Decoding Strategies in Large Language Models
liked
a Space
7 days ago
dlouapre/eiffel-tower-llama
liked
a model
10 days ago
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
Organizations
upvoted
an
article
3 days ago
liked
a
Space
7 days ago
liked
a
model
10 days ago
liked
2
models
11 days ago
upvoted
an
article
13 days ago
Article
New in llama.cpp: Model Management
•
93
reacted to
ovi054's
post with 🚀🔥
14 days ago
Post
2300
Z-Image Turbo + LoRA ⚡
ovi054/Z-Image-LORA
Z-Image Turbo is the No. 1 trending Text-to-Image model right now. You can add a custom LoRA and generate images with this Space.
👉 Try it now: ovi054/Z-Image-LORA
ovi054/Z-Image-LORA
Z-Image Turbo is the No. 1 trending Text-to-Image model right now. You can add a custom LoRA and generate images with this Space.
👉 Try it now: ovi054/Z-Image-LORA
- 3 replies
reacted to
their
post with 🔥
14 days ago
Post
2290
Muon vs MuonClip vs Muon+Adamw
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fine‑tuning? We ran head‑to‑head tests on Qwen3‑4B (10k+ high‑quality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradient‑norm spikes made training unstable. MuonClip (Kimi K2’s clipping) stabilizes long pretraining runs, yet in our small‑scale fine‑tune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muon’s spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fine‑tuning? We ran head‑to‑head tests on Qwen3‑4B (10k+ high‑quality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradient‑norm spikes made training unstable. MuonClip (Kimi K2’s clipping) stabilizes long pretraining runs, yet in our small‑scale fine‑tune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muon’s spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
liked
2
models
14 days ago
commented on
Muon vs MuonClip vs Muon+AdamW for Fine-Tuning
14 days ago
I think so. I'll test it out, try to cover that in next blog.
commented on
Muon vs MuonClip vs Muon+AdamW for Fine-Tuning
14 days ago
The performance difference b/w AdamW 32‑bit and 8‑bit is very negligible, so the 8‑bit version will essentially match (or even slightly outperform) the full‑precision AdamW. I’m not as certain about the 4‑bit variant in my tests in small training it converges comparably, but in my longer‑run experiments, it tends to fall behind.
posted
an
update
14 days ago
Post
2290
Muon vs MuonClip vs Muon+Adamw
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fine‑tuning? We ran head‑to‑head tests on Qwen3‑4B (10k+ high‑quality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradient‑norm spikes made training unstable. MuonClip (Kimi K2’s clipping) stabilizes long pretraining runs, yet in our small‑scale fine‑tune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muon’s spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
Muon has gone from an experiment to a mainstream optimizer, but does it hold up for fine‑tuning? We ran head‑to‑head tests on Qwen3‑4B (10k+ high‑quality instruction rows) to find out.
Short story: Pure Muon converged fastest at the start, but its gradient‑norm spikes made training unstable. MuonClip (Kimi K2’s clipping) stabilizes long pretraining runs, yet in our small‑scale fine‑tune it underperformed, lower token accuracy and slower convergence. The winner was the hybrid: Muon for 2D layers + AdamW for 1D layers. It delivered the best balance of stability and final performance and even beat vanilla AdamW.
Takeaway: for small-scale fine-tuning, hybrid = practical and reliable.
Next Step: scale to larger models/datasets to see if Muon’s spikes become catastrophic or if clipping wins out.
Full Blog Link: https://huggingface.co/blog/KingNish/optimizer-part1
reacted to
their
post with 🔥
14 days ago
Post
2435
I tested Muon vs MuonClip vs Muon+AdamW for fine-tuning LLMs
Just published a blog on that, Read here 👉 https://huggingface.co/blog/KingNish/optimizer-part1
Just published a blog on that, Read here 👉 https://huggingface.co/blog/KingNish/optimizer-part1
- 1 reply
posted
an
update
16 days ago
Post
2435
I tested Muon vs MuonClip vs Muon+AdamW for fine-tuning LLMs
Just published a blog on that, Read here 👉 https://huggingface.co/blog/KingNish/optimizer-part1
Just published a blog on that, Read here 👉 https://huggingface.co/blog/KingNish/optimizer-part1
-
- 1 reply
upvoted
an
article
16 days ago
Article
Muon vs MuonClip vs Muon+AdamW for Fine-Tuning
•
13
published
an
article
16 days ago
Article
Muon vs MuonClip vs Muon+AdamW for Fine-Tuning
•
13
updated
a
dataset
18 days ago
upvoted
an
article
18 days ago
Article
Tensor Parallelism (TP) in Transformers: 5 Minutes to Understand
•
62