Update README.md
Browse files
README.md
CHANGED
|
@@ -32,35 +32,42 @@ pip install xfuser>=0.4.3 absl-py peft lightning pandas deepspeed wandb av
|
|
| 32 |
|
| 33 |
### Model Preparation
|
| 34 |
```
|
| 35 |
-
pip install -U "huggingface_hub[cli]"
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
```
|
| 40 |
|
| 41 |
After proper preparation, the directory looks like:
|
| 42 |
```
|
| 43 |
./model_zoo
|
| 44 |
-
-
|
| 45 |
-
-
|
|
|
|
|
|
|
|
|
|
| 46 |
- base.safetensors
|
| 47 |
- lora.safetensors
|
| 48 |
```
|
|
|
|
| 49 |
### Usage Example
|
| 50 |
#### I2V with Motion Intensity Levels
|
| 51 |
```
|
| 52 |
-
# make sure you are in ~/LivePhoto-Wan2.1/PusaV1
|
| 53 |
CUDA_VISIBLE_DEVICES=0 python examples/pusavideo/wan_14b_multi_frames_pusa.py \
|
| 54 |
--image_paths "./demos/input_image.jpg" \
|
| 55 |
--prompt "A cute orange kitten with big round eyes stands upright on its hind legs on a smooth wooden floor. The kitten begins to move its tiny front paws up and down rhythmically, swaying its body left and right as if dancing. Its fluffy tail flicks slightly behind it, and the soft lighting creates a warm, cozy indoor atmosphere. The kitten’s ears twitch gently as it keeps its balance, adding to the charm of its playful little dance. The background stays softly blurred, keeping focus on the kitten’s adorable movements." \
|
| 56 |
--cond_position "0" \
|
| 57 |
--noise_multipliers "0" \
|
| 58 |
-
--dit_path "./model_zoo/Wan2.1/base.safetensors" \
|
| 59 |
--lora_path "./model_zoo/Wan2.1/lora.safetensors" \
|
| 60 |
--lora_alpha 1.2 \
|
| 61 |
--num_inference_steps 30 \
|
| 62 |
-
--
|
|
|
|
| 63 |
```
|
| 64 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
### Acknowledgement
|
| 66 |
-
This version is
|
|
|
|
| 32 |
|
| 33 |
### Model Preparation
|
| 34 |
```
|
| 35 |
+
pip install -U "huggingface_hub[cli]==0.34.0"
|
| 36 |
+
hf download Wan-AI/Wan2.1-T2V-1.3B Wan2.1_VAE.pth models_t5_umt5-xxl-enc-bf16.pth --local-dir ./model_zoo/Wan2.1/base/
|
| 37 |
+
hf download Wan-AI/Wan2.1-T2V-1.3B --include="google/*" --local-dir ./model_zoo/Wan2.1/base
|
| 38 |
+
hf download shirley430316/LivePhoto-Wan2.1 lora.safetensors base.safetensors --local-dir ./model_zoo/Wan2.1/
|
| 39 |
```
|
| 40 |
|
| 41 |
After proper preparation, the directory looks like:
|
| 42 |
```
|
| 43 |
./model_zoo
|
| 44 |
+
- Wan2.1
|
| 45 |
+
- base
|
| 46 |
+
- Wan2.1_VAE.pth
|
| 47 |
+
- models_t5_umt5-xxl-enc-bf16.pth
|
| 48 |
+
- google
|
| 49 |
- base.safetensors
|
| 50 |
- lora.safetensors
|
| 51 |
```
|
| 52 |
+
|
| 53 |
### Usage Example
|
| 54 |
#### I2V with Motion Intensity Levels
|
| 55 |
```
|
|
|
|
| 56 |
CUDA_VISIBLE_DEVICES=0 python examples/pusavideo/wan_14b_multi_frames_pusa.py \
|
| 57 |
--image_paths "./demos/input_image.jpg" \
|
| 58 |
--prompt "A cute orange kitten with big round eyes stands upright on its hind legs on a smooth wooden floor. The kitten begins to move its tiny front paws up and down rhythmically, swaying its body left and right as if dancing. Its fluffy tail flicks slightly behind it, and the soft lighting creates a warm, cozy indoor atmosphere. The kitten’s ears twitch gently as it keeps its balance, adding to the charm of its playful little dance. The background stays softly blurred, keeping focus on the kitten’s adorable movements." \
|
| 59 |
--cond_position "0" \
|
| 60 |
--noise_multipliers "0" \
|
|
|
|
| 61 |
--lora_path "./model_zoo/Wan2.1/lora.safetensors" \
|
| 62 |
--lora_alpha 1.2 \
|
| 63 |
--num_inference_steps 30 \
|
| 64 |
+
--cfg_scale 5 \
|
| 65 |
+
--motion_intensity 6 # valid motion intensity levels from 1 through 6
|
| 66 |
```
|
| 67 |
|
| 68 |
+
### Limitations
|
| 69 |
+
- Video generation quality is inherently limited by the capabilities of the base Wan2.1-T2V-1.3B model, e.g., camera motion control is not currently supported.
|
| 70 |
+
- Certain image types are likely to generate low quality videos, e.g. cartoon and animated contents, possibly due to dataset biases.
|
| 71 |
+
|
| 72 |
### Acknowledgement
|
| 73 |
+
This version is developed upon the codebase of [Pusa-VidGen](https://github.com/Yaofang-Liu/Pusa-VidGen). Much appreciation for the insightful project.
|