What hardware did you use to apply heretic?
#1
by
rejectallpriors
- opened
I saw a pr on GitHub with some discussion of single gpu / multi-gpu setups for running heretic against this model. What hardware did you use? What single GPU setups work for this and what configurations were required?
Thanks!
Hey,
I don't think there's a single GPU that could handle this on its own, as in BF16 the model takes up roughly 280 GB.
With heretic you also want a good batch size to make the process faster and not have it take days. I think it peaked at 810 GB of VRAM usage and took roughly 3.5 hours on 8x H200. v2 was running for 10 hours on 8x RTX PRO 6000 at a batch size of 100 with 400 trials.
good info. thanks!
kldzj
changed discussion status to
closed