Possibility of adapting to streaming video?
Thank you very much for your work. I noticed that you employ a hybrid approach combining local full attention with distant DeltaNet linear attention. Having reviewed the code of quite a few world models and streaming video models, I’ve observed that they typically do not use DeltaNet for memory management, but instead rely on various memory management strategies. Have you ever experimented with using DeltaNet for streaming video models?
Yes, this might work well for autoregressive generation settings. Unfortunately, we don't have enough bandwidth to test it :)
I noticed SANA-World use Gated DeltaNet. But every three GDN with one softmax block which means pure DeltaNet structure may not work well.
I think hybrid arch is needed (same as what people do for LLMs)