Unified-Reward-7B

We are actively gathering feedback from the community to improve our models. We welcome your input and encourage you to stay updated through our repository!!

[2025/4/15] 🔥🔥 We updated the UnifiedReward-7B to enhance its generalization and performance, incorporating valuable feedback from the community.

Model Summary

Unified-Reward-7b is the first unified reward model for multimodal understanding and generation assessment, enabling both pairwise ranking and pointwise scoring, which can be employed for vision model preference alignment.

For further details, please refer to the following resources:

📰 Paper: https://arxiv.org/pdf/2503.05236
🪐 Project Page: https://codegoat24.github.io/UnifiedReward/
🤗 Model Collections: https://huggingface.co/collections/CodeGoat24/unifiedreward-models-67c3008148c3a380d15ac63a
🤗 Dataset Collections: https://huggingface.co/collections/CodeGoat24/unifiedreward-training-data-67c300d4fd5eff00fa7f1ede
👋 Point of Contact: Yibin Wang

🔥 News

[2025/10/23] 🔥🔥🔥 We release UnifiedReward-Edit-[3b/7b/32b], a unified reward model for both Text-to-Image and Image-to-Image generation trained on approximately 700K unified image generation and editing reward data!! For image editing reward task, our models support:

Pairwise Rank — directly judge which of two edited images is better.

Pairwise Score — assign a separate score to each image in a pair.

Pointwise Score — rate a single image on two axes: instruction-following and overall image quality.

🚀 The image editing reward inference code is available at UnifiedReward-Edit/ directory, while T2I inference code is unchanged from previous models. The editing training data is preprocessed from EditScore and EditReward and will be released soon. We sincerely appreciate all contributors!!

[2025/9/25] 🔥🔥🔥 We release UnifiedReward-2.0-qwen-[3b/7b/32b/72b]. This version introduces several new capabilities:

Pairwise scoring for image and video generation assessment on Alignment, Coherence, Style dimensions.

Pointwise scoring for image and video generation assessment on Alignment, Coherence/Physics, Style dimensions.

The added inference code is available at inference_qwen/UnifiedReward-2.0-inference directory. The newly added training data has been released here 😊.

🏁 Compared with Current Reward Models

Reward Model	Method	Image Generation	Image Understanding	Video Generation	Video Understanding
PickScore	Point	√
HPS	Point	√
ImageReward	Point	√
LLaVA-Critic	Pair/Point		√
IXC-2.5-Reward	Pair/Point		√		√
VideoScore	Point			√
LiFT	Point			√
VisionReward	Point	√		√
VideoReward	Point			√
UnifiedReward (Ours)	Pair/Point	√	√	√	√

Quick Start

All pair rank and point score inference codes are provided in our github.

We take image understanding assessment as example here:

# pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git
from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN, IGNORE_INDEX
from llava.conversation import conv_templates, SeparatorStyle

from PIL import Image
import requests
import copy
import torch

import sys
import warnings
import os


warnings.filterwarnings("ignore")
pretrained = "CodeGoat24/UnifiedReward-7b"
model_name = "llava_qwen"
device = "cuda"
device_map = "auto"
tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map)  # Add any other thing you want to pass in llava_model_args

model.eval()

url = "https://github.com/LLaVA-VL/blog/blob/main/2024-10-03-llava-critic/static/images/critic_img_seven.png?raw=True"
image = Image.open(requests.get(url, stream=True).raw)
image_tensor = process_images([image], image_processor, model.config)
image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]

conv_template = "qwen_1_5"  # Make sure you use correct chat template for different models

# pairwise ranking
critic_prompt = "Given an image and a corresponding question, please serve as an unbiased and fair judge to evaluate the quality of the answers provided by a Large Multimodal Model (LMM). Determine which answer is better and explain your reasoning with specific details. Your task is provided as follows:\nQuestion: [What this image presents?]\nThe first response: [The image is a black and white sketch of a line that appears to be in the shape of a cross. The line is a simple and straightforward representation of the cross shape, with two straight lines intersecting at a point.]\nThe second response: [This is a handwritten number seven.]\nASSISTANT:\n"

# pointwise scoring
# critic_prompt = "Given an image and a corresponding question, please serve as an unbiased and fair judge to evaluate the quality of answer answers provided by a Large Multimodal Model (LMM). Score the response out of 100 and explain your reasoning with specific details. Your task is provided as follows:\nQuestion: [What this image presents?]\nThe LMM response: [This is a handwritten number seven.]\nASSISTANT:\n "

question = DEFAULT_IMAGE_TOKEN + "\n" + critic_prompt
conv = copy.deepcopy(conv_templates[conv_template])
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt_question = conv.get_prompt()

input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
image_sizes = [image.size]


cont = model.generate(
    input_ids,
    images=image_tensor,
    image_sizes=image_sizes,
    do_sample=False,
    temperature=0,
    max_new_tokens=4096,
)
text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
print(text_outputs[0])

Citation

@article{unifiedreward,
  title={Unified reward model for multimodal understanding and generation},
  author={Wang, Yibin and Zang, Yuhang and Li, Hao and Jin, Cheng and Wang, Jiaqi},
  journal={arXiv preprint arXiv:2503.05236},
  year={2025}
}