hyperflow size becomes bigger
#13
by ZKong - opened
i saw rank larger, some becomes 768 from 256. can you tell us the reason
That 768 isn't a rank increase it's the QKV fusion
- Original hyperflow file (Diffusers): attn.to_q, to_k, to_v are three separate modules, each rank 256.
- ComfyUI format: H3's attention is one fused qkv_proj layer, so the three adapters are merged into a single one: rank = 256 (Q) + 256 (K) + 256 (V) = 768. The A matrices stack (768 rows) and B becomes block-diagonal (each third of the output rows only reads its own 256 columns), so Q, K, V still can't mix it behaves exactly like three rank-256 adapters.
You'd see the same 768 in the full converted file, not just the resized one it was there from the moment we fused for ComfyUI. Same story in the rank-20 files: each of Q/K/V got its own independent rank (avg ~20.5, so the fused module ends up around 60ish total, varying per block).
The "2 .. 125" range in the resize report is the opposite phenomenon per-projection ranks shrinking adaptively: projections carrying little of the update get rank 2–8, important ones keep up to 125.
drbaph changed discussion status to closed