StarVLA/WM4A-Wan2d2-OFT-LIBERO-4in1

WM4A Wan2.2-OFT for LIBERO 4-in-1

A Vision-Language-Action (VLA) model from the StarVLA project, built on Wan2.2-TI2V-5B (a large-scale text-to-video diffusion model) as the visual backbone, driving a lightweight MLP action head (WanOFT). The model is trained on the full LIBERO 4-in-1 benchmark (libero_10 + libero_goal + libero_object + libero_spatial combined).

WanOFT is StarVLA's architecture that leverages the rich spatiotemporal features of the Wan 2.2 video diffusion model as visual representations, paired with a simple yet effective MLP action head:

1. Wan2.2 visual features — last-layer activations of Wan2.2-TI2V-5B-Diffusers provide high-quality, motion-aware visual tokens that encode dynamics well-suited for manipulation policy learning.
2. MLP action head (OFT-style) — a residual MLP action head (hidden dim 3072) produces action predictions directly from the Wan2.2 visual features and instruction tokens, offering fast inference with minimal overhead.
3. Language conditioning via instruction tokens — the task instruction is tokenised and concatenated with the visual tokens before the MLP head; no separate VLM backbone is used.

---

Model Summary

Architecture	WanOFT (Wan2.2 visual backbone + MLP action head)
Visual backbone	Wan2.2-TI2V-5B-Diffusers
Action head	MLP (hidden dim 3072, OFT-style)
Action chunk	8 steps (+ 7 future-window steps)
Action / state dim	7 / 7 (action_type: delta_qpos)
Image resolution	224 × 224, single 3rd-person view
Checkpoint license	Not independently specified; see licensing note below
Codebase	starVLA/starVLA

---

Training Data

LIBERO 4-in-1 mixture (libero_all) — all four LIBERO task suites combined into a single training stream:

Suite	Tasks	Description
libero_10	10	Long-horizon tabletop manipulation
libero_goal	10	Goal-conditioned rearrangement
libero_object	10	Object-centric pick-and-place
libero_spatial	10	Spatially varied placement

• Action representation: 7D delta_qpos, as recorded by config.full.yaml
• Image observation: single primary RGB view, resized to 224 × 224
• Per-dataset normalisation statistics are stored in dataset_statistics.json.

---

Training Recipe

Total steps	800,000 configured; only the 60k weight is released
Warm-up steps	5,000
Per-device batch size	8
Hardware	GPU type/count is not recorded in the portable config
Precision	bf16, mixed-precision
Attention impl.	SDPA
Optimizer	AdamW (β₁ = 0.9, β₂ = 0.95, ε = 1e-8, wd = 1e-8)
LR (base / VLM)	2.5e-5
LR (action head)	1e-4
LR scheduler	cosine_with_min_lr (min lr 1e-6)
Gradient clipping	1.0
Frozen modules	Public config and archived launcher disagree; confirm before reproducing

The saved recipe is preserved in config.yaml and config.full.yaml. Both contain environment-local paths, and the archived launcher contains cluster-specific settings. Audit and sanitize those files rather than running the launcher unchanged.

---

Evaluation — LIBERO 4-in-1

Following the standard LIBERO evaluation protocol (50 trials per task per suite). Numbers are success rates (↑).

Step	libero_10	libero_goal	libero_object	libero_spatial	Avg (4 suites)
10k	0.364	0.772	0.986	0.808	0.732
20k	0.750	0.900	0.942	0.896	0.872
30k	0.722	0.920	0.978	0.882	0.876
40k	0.788	0.934	0.978	0.872	0.893
50k	0.772	0.924	0.978	0.864	0.885
60k	0.860	0.954	0.978	0.874	0.916

Best checkpoint: steps_60000_pytorch_model.pt — avg 91.6 % across all four LIBERO suites.

For comparison with other StarVLA frameworks see the StarVLA Model Zoo.

---

Repository layout

.
├── README.md
├── config.yaml
├── config.full.yaml
├── dataset_statistics.json
├── summary.jsonl
├── run_libero_train.sh       # archived; audit and sanitize before use
├── checkpoints/
│   └── steps_60000_pytorch_model.pt
├── logs/
│   ├── libero_10/            # evaluation logs for 10k–60k
│   ├── libero_goal/
│   ├── libero_object/
│   └── libero_spatial/
└── wandb/                    # archived training logs

Only the 60k weight is downloadable. The 10k–50k step names refer to evaluation logs, not to released checkpoint files.

---

Download and evaluation

huggingface-cli download StarVLA/WM4A-Wan2d2-OFT-LIBERO-4in1 \
  --local-dir WM4A-Wan2d2-OFT-LIBERO-4in1

CKPT="$PWD/WM4A-Wan2d2-OFT-LIBERO-4in1/checkpoints/steps_60000_pytorch_model.pt"
python deployment/model_server/server_policy.py \
  --ckpt_path "$CKPT" --port 6694 --use_bf16 \
  --config_override framework.world_model.base_wm=Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --config_override framework.qwenvl.base_vlm=Wan-AI/Wan2.2-TI2V-5B-Diffusers

Before starting the server, map framework.world_model.base_wm in the saved config to a usable Wan-AI/Wan2.2-TI2V-5B-Diffusers snapshot. The framework.qwenvl field is a legacy compatibility field, not a separate Qwen VLM used by this architecture.

For the simulator side, follow the StarVLA LIBERO workflow.

---

Intended Use & Limitations

Intended use. Research on vision-language-action models, LIBERO tabletop manipulation benchmarks, and as a baseline for large video diffusion model features in VLA architectures.

Out-of-scope / limitations. This model is trained exclusively on LIBERO simulation data with the saved 7D delta_qpos contract. Real-robot transfer and cross-embodiment generalisation have not been evaluated. Performance may degrade on out-of-distribution scenes, objects, or instructions not present in the LIBERO training split.

Evidence and evaluation boundary

• Architecture and training fields were checked against the packaged configs and the StarVLA WanOFT implementation; the artifact list was checked against the Hub file tree.
• The reported LIBERO values are backed by the packaged per-suite logs.
• The repository releases one weight (60k), even though evaluation logs refer to 10k–50k checkpoints.
• GPU type/count and a portable base-world-model path are not recorded in the public config. The saved config and archived launcher also disagree about freeze_modules, so exact trainability must be confirmed before reproduction.
• Results cover LIBERO simulation only. Real-robot transfer, cross-embodiment generalization, and safety have not been established.
• The StarVLA code license and the upstream Wan Apache-2.0 declaration do not by themselves establish the release terms of this derived checkpoint. The checkpoint's license requires maintainer confirmation.