LoRA Training
Hello, I was wondering how people are training LoRA on this model? Is there a tool which supports it?
Hi, at the moment you can use these repositories to train LoRA for this model
https://github.com/ostris/ai-toolkit
https://github.com/bghira/SimpleTuner
https://github.com/kohya-ss/sd-scripts/tree/sd3
Do you have a configuration?
In my opinion, there isn't one best setting; it depends on your dataset. For the dimension of LoRA, choose a value around 8-128, and set alpha equal to dim/2.
I'm just scared that my lora will break XD
Just as a heads up, I tried training a lora for NetaYume with ai toolkit and am having some problems actually using it post-training. The samples generated by ai toolkit showed solid progress so the training itself seems fine. But I haven't actually been able to load the lora post-training. I assume it's just a problem with key names but you might want to try a very small test run with whatever training framework you decide on, verify that the resulting lora loads with whatever you're using for image generation, and then move on to an actual training run only after you've been able to verify it.
Hi @Portias600kNeurons ,if you want to use lora which was trained from aitoolkit you need to convert it into comfyui format. I prepared a script to convert between these format you can find it here: https://huggingface.co/duongve/NetaYume-Lumina-Image-2.0/blob/main/Script_Lora_Convert/Convert_lora_format_between_comfyui_diffusers.py
@duongve
, huge thanks for the help! Sadly, the script didn't work for me. I wind up with an error of "state_dict should be empty at this point" and then a huge list of key names. I feel bad using up too much of your time with anything that I might be able to figure out from here. Or that might be just me not understanding what I'm doing. So I'm going to try some basic testing this weekend to rule out as much of my own possible user errors, quirks of the software I'm using, etc. Then try to use your script as a guide to what might be going on with my lora. And....almost certainly run back here to beg for help after I fail.
@duongve Happily, no need, I got it working! Or rather you got it working. I'm a bit embarrassed to admit that I think the script you posted would have worked if...I apparently hadn't accidentally wound up using a version of the lora that I'd already tried to fix...which had mangled the key names. Or at least I think that's how I'd messed the lora I'd initially tried with your script. I didn't realize it until I started building on the example generation script you have on this model's main description, putting in the lora loading without a frontend silently failing on me this time, and then....egg on face - I realized my mistake. But in the end lora converted with your script, loads up fine, and verified working. But thanks again for the help and the great work you're doing!
I'm not sure how to use it though 😅
@Pajamawatso , you download the script and run it on python environment. It will automatic convert from diffusers format to comfyui