smol-training-playbook

Running on CPU Upgrade

Troubleshooting Interconnect: Share Your Experience

by nouamanetazi - opened 5 days ago

Hugging Face Smol Models Research org 5 days ago

Hi everyone! 👋

The Troubleshooting Interconnect section has some initial findings on common NCCL performance issues (CPU affinity, network topology, environment variables, container configs).

[Read the full section here →] https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook#troubleshooting-interconnect

We'd love to make this a living community resource! Have you run into:

NCCL performance bottlenecks that were tricky to debug?
Cloud-specific networking issues (AWS EFA, GCP, Azure)?
Container configuration gotchas?
Effective debugging workflows or tools?

Share your troubleshooting stories, solutions, or questions below.
Your experience could help others avoid hours of debugging! 🤗

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment