Upload folder using huggingface_hub

Eagle2-2B/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

Eagle2-2B/README.md

@@ -0,0 +1,576 @@
+---
+license: cc-by-nc-4.0
+pipeline_tag: image-text-to-text
+library_name: transformers
+base_model:
+  - google/paligemma-3b-mix-448
+  - Qwen/Qwen2.5-1.5B-Instruct
+  - google/siglip-so400m-patch14-384
+base_model_relation: merge
+language:
+  - multilingual
+tags:
+  - eagle
+  - VLM
+---
+
+
+# Eagle-2
+
+[\[📂 GitHub\]](https://github.com/NVlabs/EAGLE)   [\[📜 Eagle2 Tech Report\]](http://arxiv.org/abs/2501.14818)
+[\[🗨️ Chat Demo\]](http://eagle-vlm.xyz/)  [\[🤗 HF Demo\]](TODO)  
+## Introduction
+
+We are thrilled to release our latest Eagle2 series Vision-Language Model. Open-source Vision-Language Models (VLMs) have made significant strides in narrowing the gap with proprietary models. However, critical details about data strategies and implementation are often missing, limiting reproducibility and innovation. In this project, we focus on VLM post-training from a data-centric perspective, sharing insights into building effective data strategies from scratch. By combining these strategies with robust training recipes and model design, we introduce Eagle2, a family of performant VLMs. Our work aims to empower the open-source community to develop competitive VLMs with transparent processes.
+
+
+
+In this repo, we are open-sourcing Eagle2-2B, a lightweight model that achieves remarkable efficiency and speed while maintaining solid performance.
+
+
+
+
+
+
+
+
+## Model Zoo
+We provide the following models:
+
+| model name         | LLM  | Vision  | Max Length| HF Link|
+| ----------- | ------- |---------|-|-|
+| Eagle2-1B | [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) |  Siglip    | 16K| [🤗 link](https://huggingface.co/NVIDIA/Eagle2-1B)|
+| Eagle2-2B | [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) |  Siglip    | 16K| [🤗 link](https://huggingface.co/NVIDIA/Eagle2-2B)|
+| Eagle2-9B | [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) |  Siglip+ConvNext    | 16K| [🤗 link](https://huggingface.co/NVIDIA/Eagle2-9B)|
+
+## Benchmark Results
+|          Benchmark           | InternVL2-2B | InternVL2.5-2B | InternVL2-4B |Qwen2-VL-2B| Eagle2-2B|
+| :--------------------------: | :------------------: | :----------------: | :----------: |:----------: |:----------: |  
+|    DocVQA<sub>test</sub>     |         86.9 |        88.7  |     89.2    |90.1|88.0|
+|    ChartQA<sub>test</sub>    |          76.2        |        79.2        |     81.5     |73.0|82.0|
+|    InfoVQA<sub>test</sub>    |         58.9   |        60.9     |    67.0     |65.5|65.8|
+|    TextVQA<sub>val</sub>     |         73.4         |        74.3       |     74.4   |79.7|79.1|
+|           OCRBench           |         784          |        804         |     788      |809|818|
+|      MME<sub>sum</sub>       |   1876.8         |    2138.2    |    2059.8   |1872.0  | 2109.8
+|         RealWorldQA          |        57.3     |        60.1      |    60.7     |62.6|63.1|
+|     AI2D<sub>test</sub>      |        74.1   |        74.9     |     74.7    | 78.9 |79.3|
+|      MMMU<sub>val</sub>      |          36.3       |    43.6     | 47.9  |41.1|43.1|
+| MMVet<sub>GPT-4-Turbo</sub>  |         39.5       |       60.8       |     51.0     | 49.5|53.8|
+|   HallBench<sub>avg</sub>    |         37.9      |        42.6      |     41.9     |41.7|45.8
+| MathVista<sub>testmini</sub> |         46.3      |        51.3       |     58.6     |43.0|54.7|
+| MMstar |            50.1    |       53.7     |     54.3|48.0|56.4|
+
+
+
+## Quick Start
+
+
+
+We provide a [demo inference script](./demo.py) to help you quickly start using the model. We support different input types: 
+- pure text input
+- single image input
+- multiple image input
+- video input
+
+### 0. Install the dependencies
+
+```bash
+pip install transformers
+pip install flash-attn
+```
+**Note**: Latest version of transformers is not compatible with the model.
+
+### 1. Prepare the Model worker
+
+<details>
+  <summary>Click to expand</summary>
+
+```python
+
+"""
+A model worker executes the model.
+Copied and modified from https://github.com/OpenGVLab/InternVL/blob/main/streamlit_demo/model_worker.py
+"""
+# Importing torch before transformers can cause `segmentation fault`
+from transformers import AutoModel, AutoTokenizer, TextIteratorStreamer, AutoConfig
+
+import argparse
+import base64
+import json
+import os
+import decord
+import threading
+import time
+from io import BytesIO
+from threading import Thread
+import math
+import requests
+import torch
+import torchvision.transforms as T
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+import numpy as np
+
+
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+
+SIGLIP_MEAN = (0.5, 0.5, 0.5)
+SIGLIP_STD = (0.5, 0.5, 0.5)
+
+
+def get_seq_frames(total_num_frames, desired_num_frames=-1, stride=-1):
+    """
+    Calculate the indices of frames to extract from a video.
+
+    Parameters:
+    total_num_frames (int): Total number of frames in the video.
+    desired_num_frames (int): Desired number of frames to extract.
+
+    Returns:
+    list: List of indices of frames to extract.
+    """
+    
+    assert desired_num_frames > 0 or stride > 0 and not (desired_num_frames > 0 and stride > 0)
+
+    if stride > 0:
+        return list(range(0, total_num_frames, stride))
+    
+    # Calculate the size of each segment from which a frame will be extracted
+    seg_size = float(total_num_frames - 1) / desired_num_frames
+
+    seq = []
+    for i in range(desired_num_frames):
+        # Calculate the start and end indices of each segment
+        start = int(np.round(seg_size * i))
+        end = int(np.round(seg_size * (i + 1)))
+
+        # Append the middle index of the segment to the list
+        seq.append((start + end) // 2)
+
+    return seq
+
+def build_video_prompt(meta_list, num_frames, time_position=False):
+    # if time_position is True, the frame_timestamp is used.
+    # 1. pass time_position, 2. use env TIME_POSITION
+    time_position = os.environ.get("TIME_POSITION", time_position)
+    prefix = f"This is a video:\n"
+    for i in range(num_frames):
+        if time_position:
+            frame_txt = f"Frame {i+1} sampled at {meta_list[i]:.2f} seconds: <image>\n"
+        else:
+            frame_txt = f"Frame {i+1}: <image>\n"
+        prefix += frame_txt
+    return prefix
+
+def load_video(video_path, num_frames=64, frame_cache_root=None):
+    if isinstance(video_path, str):
+        video = decord.VideoReader(video_path)
+    elif isinstance(video_path, dict):
+        assert False, 'we not support vidoe: "video_path" as input'
+    fps = video.get_avg_fps()
+    sampled_frames = get_seq_frames(len(video), num_frames)
+    samepld_timestamps = [i / fps for i in sampled_frames]
+    frames = video.get_batch(sampled_frames).asnumpy()
+    images = [Image.fromarray(frame) for frame in frames]
+    
+    return images, build_video_prompt(samepld_timestamps, len(images), time_position=True)
+
+def load_image(image):
+    if isinstance(image, str) and os.path.exists(image):
+        return Image.open(image)
+    elif isinstance(image, dict):
+        if 'disk_path' in image:
+            return Image.open(image['disk_path'])
+        elif 'base64' in image:
+            return Image.open(BytesIO(base64.b64decode(image['base64'])))
+        elif 'url' in image:
+            response = requests.get(image['url'])
+            return Image.open(BytesIO(response.content))
+        elif 'bytes' in image:
+            return Image.open(BytesIO(image['bytes']))
+        else:
+            raise ValueError(f'Invalid image: {image}')
+    else:
+        raise ValueError(f'Invalid image: {image}')
+
+def build_transform(input_size, norm_type='imagenet'):
+    if norm_type == 'imagenet':
+        MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
+    elif norm_type == 'siglip':
+        MEAN, STD = SIGLIP_MEAN, SIGLIP_STD
+        
+    transform = T.Compose([
+        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(mean=MEAN, std=STD)
+    ])
+    return transform
+
+
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    """
+    previous version mainly foucs on ratio.
+    We also consider area ratio here.
+    """
+    best_factor = float('-inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
+        area_ratio = (ratio[0]*ratio[1]*image_size*image_size)/ area
+        """
+        new area > 60% of original image area is enough.
+        """
+        factor_based_on_area_n_ratio = min((ratio[0]*ratio[1]*image_size*image_size)/ area, 0.6)* \
+                                     min(target_aspect_ratio/aspect_ratio, aspect_ratio/target_aspect_ratio)
+        
+        if factor_based_on_area_n_ratio > best_factor:
+            best_factor = factor_based_on_area_n_ratio
+            best_ratio = ratio
+        
+    return best_ratio
+
+
+def dynamic_preprocess(image, min_num=1, max_num=6, image_size=448, use_thumbnail=False):
+    orig_width, orig_height = image.size
+    aspect_ratio = orig_width / orig_height
+
+    # calculate the existing image aspect ratio
+    target_ratios = set(
+        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+        i * j <= max_num and i * j >= min_num)
+    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+
+    # find the closest aspect ratio to the target
+    target_aspect_ratio = find_closest_aspect_ratio(
+        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+
+    # calculate the target width and height
+    target_width = image_size * target_aspect_ratio[0]
+    target_height = image_size * target_aspect_ratio[1]
+    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
+
+    # resize the image
+    resized_img = image.resize((target_width, target_height))
+    processed_images = []
+    for i in range(blocks):
+        box = (
+            (i % (target_width // image_size)) * image_size,
+            (i // (target_width // image_size)) * image_size,
+            ((i % (target_width // image_size)) + 1) * image_size,
+            ((i // (target_width // image_size)) + 1) * image_size
+        )
+        # split the image
+        split_img = resized_img.crop(box)
+        processed_images.append(split_img)
+    assert len(processed_images) == blocks
+    if use_thumbnail and len(processed_images) != 1:
+        thumbnail_img = image.resize((image_size, image_size))
+        processed_images.append(thumbnail_img)
+    return processed_images
+
+def split_model(model_path, device):
+
+    device_map = {}
+    world_size = torch.cuda.device_count()
+    config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
+    num_layers = config.llm_config.num_hidden_layers
+
+    print('world_size', world_size)
+    num_layers_per_gpu_ = math.floor(num_layers / (world_size - 1))
+    num_layers_per_gpu = [num_layers_per_gpu_] * world_size
+    num_layers_per_gpu[device] = num_layers - num_layers_per_gpu_ * (world_size-1)
+    print(num_layers_per_gpu)
+    layer_cnt = 0
+    for i, num_layer in enumerate(num_layers_per_gpu):
+        for j in range(num_layer):
+            device_map[f'language_model.model.layers.{layer_cnt}'] = i
+            layer_cnt += 1
+    device_map['vision_model'] = device
+    device_map['mlp1'] = device
+    device_map['language_model.model.tok_embeddings'] = device
+    device_map['language_model.model.embed_tokens'] = device
+    device_map['language_model.output'] = device
+    device_map['language_model.model.norm'] = device
+    device_map['language_model.lm_head'] = device
+    device_map['language_model.model.rotary_emb'] = device
+    device_map[f'language_model.model.layers.{num_layers - 1}'] = device
+    return device_map
+
+class ModelWorker:
+    def __init__(self, model_path, model_name,
+                 load_8bit, device):
+
+        if model_path.endswith('/'):
+            model_path = model_path[:-1]
+        if model_name is None:
+            model_paths = model_path.split('/')
+            if model_paths[-1].startswith('checkpoint-'):
+                self.model_name = model_paths[-2] + '_' + model_paths[-1]
+            else:
+                self.model_name = model_paths[-1]
+        else:
+            self.model_name = model_name
+
+        print(f'Loading the model {self.model_name}')
+
+        tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
+        tokens_to_keep = ['<box>', '</box>', '<ref>', '</ref>']
+        tokenizer.additional_special_tokens = [item for item in tokenizer.additional_special_tokens if item not in tokens_to_keep]
+        self.tokenizer = tokenizer
+        config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
+        model_type = config.vision_config.model_type
+        self.device = torch.cuda.current_device()
+        if model_type == 'siglip_vision_model':
+            self.norm_type = 'siglip'
+        elif model_type == 'MOB':
+            self.norm_type = 'siglip'
+        else:
+            self.norm_type = 'imagenet'
+
+        if any(x in model_path.lower() for x in ['34b']):
+            device_map = split_model(model_path, self.device)
+        else:
+            device_map = None
+        
+        if device_map is not None:    
+            self.model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,
+                                               low_cpu_mem_usage=True,
+                                               device_map=device_map, 
+                                               trust_remote_code=True,
+                                               load_in_8bit=load_8bit).eval()
+        else:
+            self.model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,
+                                               trust_remote_code=True,
+                                               load_in_8bit=load_8bit).eval()  
+
+        if not load_8bit and device_map is None:
+            self.model = self.model.to(device)
+        self.load_8bit = load_8bit
+        
+        self.model_path = model_path
+        self.image_size = self.model.config.force_image_size
+        self.context_len = tokenizer.model_max_length
+        self.per_tile_len = 256
+
+    def reload_model(self):
+        del self.model
+        torch.cuda.empty_cache()
+        if self.device == 'auto':
+            os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
+            # This can make distributed deployment work properly
+            self.model = AutoModel.from_pretrained(
+                self.model_path,
+                load_in_8bit=self.load_8bit,
+                torch_dtype=torch.bfloat16,
+                device_map=self.device_map,
+                trust_remote_code=True).eval()
+        else:
+            self.model = AutoModel.from_pretrained(
+                self.model_path,
+                load_in_8bit=self.load_8bit,
+                torch_dtype=torch.bfloat16,
+                trust_remote_code=True).eval()
+        if not self.load_8bit and not self.device == 'auto':
+            self.model = self.model.cuda()
+
+    @torch.inference_mode()
+    def generate(self, params):
+        system_message = params['prompt'][0]['content']
+        send_messages = params['prompt'][1:]
+        max_input_tiles = params['max_input_tiles']
+        temperature = params['temperature']
+        top_p = params['top_p']
+        max_new_tokens = params['max_new_tokens']
+        repetition_penalty = params['repetition_penalty']
+        video_frame_num = params.get('video_frame_num', 64)
+        do_sample = True if temperature > 0.0 else False
+
+        global_image_cnt = 0
+        history, pil_images, max_input_tile_list = [], [], []
+        for message in send_messages:
+            if message['role'] == 'user':
+                prefix = ''
+                if 'image' in message:
+                    for image_data in message['image']:
+                        pil_images.append(load_image(image_data))
+                        prefix = prefix + f'<image {global_image_cnt + 1}><image>\n'
+                        global_image_cnt += 1
+                        max_input_tile_list.append(max_input_tiles)
+                if 'video' in message:
+                    for video_data in message['video']:
+                        video_frames, tmp_prefix = load_video(video_data, num_frames=video_frame_num)
+                        pil_images.extend(video_frames)
+                        prefix = prefix + tmp_prefix
+                        global_image_cnt += len(video_frames)
+                        max_input_tile_list.extend([1] * len(video_frames))
+                content = prefix + message['content']
+                history.append([content, ])
+            else:
+                history[-1].append(message['content'])
+        question, history = history[-1][0], history[:-1]
+
+        if global_image_cnt == 1:
+            question = question.replace('<image 1><image>\n', '<image>\n')
+            history = [[item[0].replace('<image 1><image>\n', '<image>\n'), item[1]] for item in history]
+
+
+        try:
+            assert len(max_input_tile_list) == len(pil_images), 'The number of max_input_tile_list and pil_images should be the same.'
+        except Exception as e:
+            from IPython import embed; embed()
+            exit()
+            print(f'Error: {e}')
+            print(f'max_input_tile_list: {max_input_tile_list}, pil_images: {pil_images}')
+            # raise e
+
+        old_system_message = self.model.system_message
+        self.model.system_message = system_message
+        
+        transform = build_transform(input_size=self.image_size, norm_type=self.norm_type)
+        if len(pil_images) > 0:
+            max_input_tiles_limited_by_contect = params['max_input_tiles']
+            while True:
+                image_tiles = []
+                for current_max_input_tiles, pil_image in zip(max_input_tile_list, pil_images):
+                    if self.model.config.dynamic_image_size:
+                        tiles = dynamic_preprocess(
+                            pil_image, image_size=self.image_size, max_num=min(current_max_input_tiles, max_input_tiles_limited_by_contect),
+                            use_thumbnail=self.model.config.use_thumbnail)
+                    else:
+                        tiles = [pil_image]
+                    image_tiles += tiles
+                if (len(image_tiles) * self.per_tile_len < self.context_len):
+                    break
+                else:
+                    max_input_tiles_limited_by_contect -= 2
+                
+                if max_input_tiles_limited_by_contect < 1:
+                    break
+                    
+            pixel_values = [transform(item) for item in image_tiles]
+            pixel_values = torch.stack(pixel_values).to(self.model.device, dtype=torch.bfloat16)
+            print(f'Split images to {pixel_values.shape}')
+        else:
+            pixel_values = None
+
+        generation_config = dict(
+            num_beams=1,
+            max_new_tokens=max_new_tokens,
+            do_sample=do_sample,
+            temperature=temperature,
+            repetition_penalty=repetition_penalty,
+            max_length=self.context_len,
+            top_p=top_p,
+        )
+
+        response = self.model.chat(
+            tokenizer=self.tokenizer,
+            pixel_values=pixel_values,
+            question=question,
+            history=history,
+            return_history=False,
+            generation_config=generation_config,
+        )
+        self.model.system_message = old_system_message
+        return {'text': response, 'error_code': 0}
+
+
+
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model-path', type=str, default='nvidia/Eagle2-2B')
+    parser.add_argument('--model-name', type=str, default='Eagle2-2B')
+    parser.add_argument('--device', type=str, default='cuda')
+    parser.add_argument('--load-8bit', action='store_true')
+    args = parser.parse_args()
+    print(f'args: {args}')
+
+    worker = ModelWorker(
+                         args.model_path,
+                         args.model_name,
+                         args.load_8bit,
+                         args.device)
+```
+</details>
+
+
+### 2. Prepare the Prompt
+
+- Single image input
+```python
+prompt = [
+        {'role': 'system', 'content': 'You are a helpful assistant.'},
+        {'role': 'user', 'content': 'Describe this image in details.', 
+            'image':[
+                {'url': 'https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/[email protected]'}
+            ],
+        }
+    ]
+```
+
+- Multiple image input
+```python
+prompt = [
+        {'role': 'system', 'content': 'You are a helpful assistant.'},
+        {'role': 'user', 'content': 'Describe these two images in details.', 
+            'image':[
+                {'url': 'https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/[email protected]'},
+                {'url': 'https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/[email protected]'}
+            ],
+        }
+    ]
+```
+
+- Video input
+```python
+prompt = [
+        {'role': 'system', 'content': 'You are a helpful assistant.'},
+        {'role': 'user', 'content': 'Describe this video in details.', 
+            'video':[
+                'path/to/your/video.mp4'
+            ],
+        }
+    ]
+```
+
+### 3. Generate the response    
+```python
+params = {
+    'prompt': prompt,
+    'max_input_tiles': 24,
+    'temperature': 0.7,
+    'top_p': 1.0,
+    'max_new_tokens': 4096,
+    'repetition_penalty': 1.0,
+    }
+worker.generate(params)
+```
+
+## TODO
+- [ ] Support vLLM Inference
+- [ ] Provide AWQ Quantization Weights
+- [ ] Provide fine-tuning scripts
+
+
+## License/Terms of Use
+- The code is released under the Apache 2.0 license as found in the [LICENSE](https://huggingface.co/NVEagle/Eagle-X5-13B-Chat/blob/main/LICENSE) file.
+- The pretrained model weights are released under the [Creative Commons Attribution: Non-Commercial 4.0 International](https://spdx.org/licenses/CC-BY-NC-4.0) <br>
+- The service is a research preview intended for non-commercial use only, and is subject to the following licenses and terms:
+  - Model License of Qwen2.5-1.5B-Instruct: [Apache-2.0](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/blob/main/LICENSE)
+  - Model License of PaliGemma: [Gemma license](https://ai.google.dev/gemma/terms)
+
+
+
+## Citation
+
+## Ethical Considerations
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.    
+
+Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
+ 

Eagle2-2B/added_tokens.json

@@ -0,0 +1,33 @@
+{
+  "</box>": 151673,
+  "</img>": 151666,
+  "</quad>": 151669,
+  "</ref>": 151671,
+  "</tool_call>": 151658,
+  "<IMG_CONTEXT>": 151667,
+  "<box>": 151672,
+  "<img>": 151665,
+  "<quad>": 151668,
+  "<ref>": 151670,
+  "<tool_call>": 151657,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

Eagle2-2B/config.json

@@ -0,0 +1,206 @@
+{
+    "_commit_hash": null,
+    "_name_or_path": "",
+    "architectures": [
+        "Eagle2ChatModel"
+    ],
+    "auto_map": {
+        "AutoConfig": "configuration_eagle_chat.Eagle2ChatConfig",
+        "AutoModel": "modeling_eagle_chat.Eagle2ChatModel",
+        "AutoModelForCausalLM": "modeling_eagle_chat.Eagle2ChatModel"
+    },
+    "downsample_ratio": 0.5,
+    "dynamic_image_size": true,
+    "efficient_loss": true,
+    "force_image_size": 448,
+    "keep_aspect_ratio": false,
+    "llm_config": {
+        "_name_or_path": "./pretrained/Qwen2_5-1_5B-Instruct",
+        "add_cross_attention": false,
+        "architectures": [
+            "Qwen2ForCausalLM"
+        ],
+        "attention_dropout": 0.0,
+        "attn_implementation": "flash_attention_2",
+        "auto_map": {
+            "AutoConfig": "configuration_qwen2.Qwen2Config",
+            "AutoModel": "modeling_qwen2.Qwen2Model",
+            "AutoModelForCausalLM": "modeling_qwen2.Qwen2ForCausalLM"
+        },
+        "bad_words_ids": null,
+        "begin_suppress_tokens": null,
+        "bos_token_id": 151643,
+        "chunk_size_feed_forward": 0,
+        "cross_attention_hidden_size": null,
+        "decoder_start_token_id": null,
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "early_stopping": false,
+        "encoder_no_repeat_ngram_size": 0,
+        "eos_token_id": 151645,
+        "exponential_decay_length_penalty": null,
+        "finetuning_task": null,
+        "forced_bos_token_id": null,
+        "forced_eos_token_id": null,
+        "hidden_act": "silu",
+        "hidden_size": 1536,
+        "id2label": {
+            "0": "LABEL_0",
+            "1": "LABEL_1"
+        },
+        "initializer_range": 0.02,
+        "intermediate_size": 8960,
+        "is_decoder": false,
+        "is_encoder_decoder": false,
+        "label2id": {
+            "LABEL_0": 0,
+            "LABEL_1": 1
+        },
+        "length_penalty": 1.0,
+        "max_length": 20,
+        "max_position_embeddings": 32768,
+        "max_window_layers": 21,
+        "min_length": 0,
+        "model_type": "qwen2",
+        "no_repeat_ngram_size": 0,
+        "num_attention_heads": 12,
+        "num_beam_groups": 1,
+        "num_beams": 1,
+        "num_hidden_layers": 28,
+        "num_key_value_heads": 2,
+        "num_return_sequences": 1,
+        "output_attentions": false,
+        "output_hidden_states": false,
+        "output_scores": false,
+        "pad_token_id": null,
+        "prefix": null,
+        "problem_type": null,
+        "pruned_heads": {},
+        "remove_invalid_values": false,
+        "repetition_penalty": 1.0,
+        "return_dict": true,
+        "return_dict_in_generate": false,
+        "rms_norm_eps": 1e-06,
+        "rope_theta": 1000000.0,
+        "sep_token_id": null,
+        "sliding_window": 32768,
+        "suppress_tokens": null,
+        "task_specific_params": null,
+        "temperature": 1.0,
+        "tf_legacy_loss": false,
+        "tie_encoder_decoder": false,
+        "tie_word_embeddings": true,
+        "tokenizer_class": null,
+        "top_k": 50,
+        "top_p": 1.0,
+        "torch_dtype": "bfloat16",
+        "torchscript": false,
+        "transformers_version": "4.37.2",
+        "typical_p": 1.0,
+        "use_bfloat16": false,
+        "use_cache": false,
+        "use_sliding_window": false,
+        "vocab_size": 151674
+    },
+    "loss_version": "v4",
+    "max_dynamic_patch": 12,
+    "min_dynamic_patch": 1,
+    "mlp_checkpoint": true,
+    "model_type": "eagle_chat",
+    "pad2square": false,
+    "pre_feature_reduction": false,
+    "ps_version": "v2",
+    "select_layer": -1,
+    "template": "qwen2-chat",
+    "torch_dtype": "bfloat16",
+    "transformers_version": null,
+    "use_backbone_lora": 0,
+    "use_llm_lora": 0,
+    "use_thumbnail": true,
+    "vision_config": {
+        "_name_or_path": "",
+        "add_cross_attention": false,
+        "architectures": [
+            "SiglipVisionModel"
+        ],
+        "attention_dropout": 0.0,
+        "auto_map": {
+            "AutoConfig": "configuration_siglip.SiglipVisionConfig",
+            "AutoModel": "modeling_siglip.SiglipVisionModel"
+        },
+        "bad_words_ids": null,
+        "begin_suppress_tokens": null,
+        "bos_token_id": null,
+        "chunk_size_feed_forward": 0,
+        "cross_attention_hidden_size": null,
+        "decoder_start_token_id": null,
+        "diversity_penalty": 0.0,
+        "do_sample": false,
+        "drop_path_rate": 0.1,
+        "early_stopping": false,
+        "encoder_no_repeat_ngram_size": 0,
+        "eos_token_id": null,
+        "exponential_decay_length_penalty": null,
+        "finetuning_task": null,
+        "forced_bos_token_id": null,
+        "forced_eos_token_id": null,
+        "hidden_act": "gelu_pytorch_tanh",
+        "hidden_size": 1152,
+        "id2label": {
+            "0": "LABEL_0",
+            "1": "LABEL_1"
+        },
+        "image_size": 448,
+        "intermediate_size": 4304,
+        "is_decoder": false,
+        "is_encoder_decoder": false,
+        "label2id": {
+            "LABEL_0": 0,
+            "LABEL_1": 1
+        },
+        "layer_norm_eps": 1e-06,
+        "length_penalty": 1.0,
+        "max_length": 20,
+        "min_length": 0,
+        "model_type": "siglip_vision_model",
+        "no_repeat_ngram_size": 0,
+        "num_attention_heads": 16,
+        "num_beam_groups": 1,
+        "num_beams": 1,
+        "num_channels": 3,
+        "num_hidden_layers": 27,
+        "num_image_tokens": 1024,
+        "num_return_sequences": 1,
+        "output_attentions": false,
+        "output_hidden_states": false,
+        "output_scores": false,
+        "pad_token_id": null,
+        "patch_size": 14,
+        "prefix": null,
+        "problem_type": null,
+        "projection_dim": 2048,
+        "projector_hidden_act": "gelu_fast",
+        "pruned_heads": {},
+        "remove_invalid_values": false,
+        "repetition_penalty": 1.0,
+        "return_dict": true,
+        "return_dict_in_generate": false,
+        "sep_token_id": null,
+        "suppress_tokens": null,
+        "task_specific_params": null,
+        "temperature": 1.0,
+        "tf_legacy_loss": false,
+        "tie_encoder_decoder": false,
+        "tie_word_embeddings": true,
+        "tokenizer_class": null,
+        "top_k": 50,
+        "top_p": 1.0,
+        "torch_dtype": "bfloat16",
+        "torchscript": false,
+        "transformers_version": "4.37.2",
+        "typical_p": 1.0,
+        "use_bfloat16": false,
+        "vision_use_head": false,
+        "_attn_implementation": "flash_attention_2"
+    }
+}

Eagle2-2B/configuration_eagle_chat.py

@@ -0,0 +1,102 @@
+# --------------------------------------------------------
+# Eagle2
+# Copyright (c) 2025 NVIDIA
+# Licensed under The Apache License [see LICENSE for details]
+# --------------------------------------------------------
+
+import copy
+
+from transformers import AutoConfig, LlamaConfig
+from transformers.configuration_utils import PretrainedConfig
+from transformers.utils import logging
+from transformers.models.siglip.configuration_siglip import SiglipVisionConfig
+from transformers.models.qwen2.configuration_qwen2 import Qwen2Config
+logger = logging.get_logger(__name__)
+
+class Eagle2ChatConfig(PretrainedConfig):
+    model_type = 'eagle_chat'
+    is_composition = True
+
+    def __init__(
+            self,
+            vision_config=None,
+            llm_config=None,
+            use_backbone_lora=0,
+            use_llm_lora=0,
+            select_layer=-1,
+            force_image_size=None,
+            downsample_ratio=0.5,
+            template=None,
+            dynamic_image_size=False,
+            use_thumbnail=False,
+            min_dynamic_patch=1,
+            max_dynamic_patch=6,
+            mlp_checkpoint=True,
+            pre_feature_reduction=False,
+            keep_aspect_ratio=False,
+            vocab_size=-1,
+            **kwargs):
+        super().__init__(**kwargs)
+
+        if vision_config is None:
+            vision_config = {}
+            logger.info('vision_config is None. Initializing Vision Encoders with default values.')
+
+        if llm_config is None:
+            llm_config = {}
+            logger.info('llm_config is None. Initializing the LLM config with default values')
+
+        if vision_config['model_type'] == 'siglip_vision_model':
+            self.vision_config = SiglipVisionConfig(**vision_config)
+        else:
+            raise ValueError('Unsupported model_type: {}'.format(vision_config['model_type']))
+
+        if llm_config['architectures'][0] == 'LlamaForCausalLM':
+            self.llm_config = LlamaConfig(**llm_config)
+        elif llm_config['architectures'][0] == 'Qwen2ForCausalLM':
+            self.llm_config = Qwen2Config(**llm_config)
+        else:
+            raise ValueError('Unsupported architecture: {}'.format(llm_config['architectures'][0]))
+        self.use_backbone_lora = use_backbone_lora
+        self.use_llm_lora = use_llm_lora
+        self.select_layer = select_layer
+        self.force_image_size = force_image_size
+        self.downsample_ratio = downsample_ratio
+        self.template = template
+        self.dynamic_image_size = dynamic_image_size
+        self.use_thumbnail = use_thumbnail
+        self.min_dynamic_patch = min_dynamic_patch
+        self.max_dynamic_patch = max_dynamic_patch
+        self.mlp_checkpoint = mlp_checkpoint
+        self.pre_feature_reduction = pre_feature_reduction
+        self.keep_aspect_ratio = keep_aspect_ratio
+        self.vocab_size = self.llm_config.vocab_size
+        logger.info(f'keep_aspect_ratio: {self.keep_aspect_ratio}')
+        logger.info(f'vision_select_layer: {self.select_layer}')
+        logger.info(f'min_dynamic_patch: {self.min_dynamic_patch}')
+        logger.info(f'max_dynamic_patch: {self.max_dynamic_patch}')
+
+    def to_dict(self):
+        """
+        Serializes this instance to a Python dictionary. Override the default [`~PretrainedConfig.to_dict`].
+
+        Returns:
+            `Dict[str, any]`: Dictionary of all the attributes that make up this configuration instance,
+        """
+        output = copy.deepcopy(self.__dict__)
+        output['vision_config'] = self.vision_config.to_dict()
+        output['llm_config'] = self.llm_config.to_dict()
+        output['model_type'] = self.__class__.model_type
+        output['use_backbone_lora'] = self.use_backbone_lora
+        output['use_llm_lora'] = self.use_llm_lora
+        output['select_layer'] = self.select_layer
+        output['force_image_size'] = self.force_image_size
+        output['downsample_ratio'] = self.downsample_ratio
+        output['template'] = self.template
+        output['dynamic_image_size'] = self.dynamic_image_size
+        output['use_thumbnail'] = self.use_thumbnail
+        output['min_dynamic_patch'] = self.min_dynamic_patch
+        output['max_dynamic_patch'] = self.max_dynamic_patch
+        output['keep_aspect_ratio'] = self.keep_aspect_ratio
+
+        return output

Eagle2-2B/demo.py

@@ -0,0 +1,428 @@
+
+"""
+A model worker executes the model.
+"""
+from transformers import AutoModel, AutoTokenizer, TextIteratorStreamer, AutoConfig
+import argparse
+import base64
+import json
+import os
+import decord
+import threading
+import time
+from io import BytesIO
+from threading import Thread
+import math
+import requests
+import torch
+import torchvision.transforms as T
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+
+import numpy as np
+
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+
+SIGLIP_MEAN = (0.5, 0.5, 0.5)
+SIGLIP_STD = (0.5, 0.5, 0.5)
+
+
+def get_seq_frames(total_num_frames, desired_num_frames=-1, stride=-1):
+    """
+    Calculate the indices of frames to extract from a video.
+
+    Parameters:
+    total_num_frames (int): Total number of frames in the video.
+    desired_num_frames (int): Desired number of frames to extract.
+
+    Returns:
+    list: List of indices of frames to extract.
+    """
+    
+    assert desired_num_frames > 0 or stride > 0 and not (desired_num_frames > 0 and stride > 0)
+
+    if stride > 0:
+        return list(range(0, total_num_frames, stride))
+    
+    # Calculate the size of each segment from which a frame will be extracted
+    seg_size = float(total_num_frames - 1) / desired_num_frames
+
+    seq = []
+    for i in range(desired_num_frames):
+        # Calculate the start and end indices of each segment
+        start = int(np.round(seg_size * i))
+        end = int(np.round(seg_size * (i + 1)))
+
+        # Append the middle index of the segment to the list
+        seq.append((start + end) // 2)
+
+    return seq
+
+def build_video_prompt(meta_list, num_frames, time_position=False):
+    # if time_position is True, the frame_timestamp is used.
+    # 1. pass time_position, 2. use env TIME_POSITION
+    time_position = os.environ.get("TIME_POSITION", time_position)
+    prefix = f"This is a video:\n"
+    for i in range(num_frames):
+        if time_position:
+            frame_txt = f"Frame {i+1} sampled at {meta_list[i]:.2f} seconds: <image>\n"
+        else:
+            frame_txt = f"Frame {i+1}: <image>\n"
+        prefix += frame_txt
+    return prefix
+
+def load_video(video_path, num_frames=64, frame_cache_root=None):
+    if isinstance(video_path, str):
+        video = decord.VideoReader(video_path)
+    elif isinstance(video_path, dict):
+        assert False, 'we not support vidoe: "video_path" as input'
+    fps = video.get_avg_fps()
+    sampled_frames = get_seq_frames(len(video), num_frames)
+    samepld_timestamps = [i / fps for i in sampled_frames]
+    frames = video.get_batch(sampled_frames).asnumpy()
+    images = [Image.fromarray(frame) for frame in frames]
+    
+    return images, build_video_prompt(samepld_timestamps, len(images), time_position=True)
+
+def load_image(image):
+    if isinstance(image, str) and os.path.exists(image):
+        return Image.open(image)
+    elif isinstance(image, dict):
+        if 'disk_path' in image:
+            return Image.open(image['disk_path'])
+        elif 'base64' in image:
+            return Image.open(BytesIO(base64.b64decode(image['base64'])))
+        elif 'url' in image:
+            response = requests.get(image['url'])
+            return Image.open(BytesIO(response.content))
+        elif 'bytes' in image:
+            return Image.open(BytesIO(image['bytes']))
+        else:
+            raise ValueError(f'Invalid image: {image}')
+    else:
+        raise ValueError(f'Invalid image: {image}')
+
+def build_transform(input_size, norm_type='imagenet'):
+    if norm_type == 'imagenet':
+        MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
+    elif norm_type == 'siglip':
+        MEAN, STD = SIGLIP_MEAN, SIGLIP_STD
+        
+    transform = T.Compose([
+        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(mean=MEAN, std=STD)
+    ])
+    return transform
+
+
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    """
+    previous version mainly foucs on ratio.
+    We also consider area ratio here.
+    """
+    best_factor = float('-inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
+        area_ratio = (ratio[0]*ratio[1]*image_size*image_size)/ area
+        """
+        new area > 60% of original image area is enough.
+        """
+        factor_based_on_area_n_ratio = min((ratio[0]*ratio[1]*image_size*image_size)/ area, 0.6)* \
+                                     min(target_aspect_ratio/aspect_ratio, aspect_ratio/target_aspect_ratio)
+        
+        if factor_based_on_area_n_ratio > best_factor:
+            best_factor = factor_based_on_area_n_ratio
+            best_ratio = ratio
+        
+    return best_ratio
+
+
+def dynamic_preprocess(image, min_num=1, max_num=6, image_size=448, use_thumbnail=False):
+    orig_width, orig_height = image.size
+    aspect_ratio = orig_width / orig_height
+
+    # calculate the existing image aspect ratio
+    target_ratios = set(
+        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+        i * j <= max_num and i * j >= min_num)
+    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+
+    # find the closest aspect ratio to the target
+    target_aspect_ratio = find_closest_aspect_ratio(
+        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+
+    # calculate the target width and height
+    target_width = image_size * target_aspect_ratio[0]
+    target_height = image_size * target_aspect_ratio[1]
+    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
+
+    # resize the image
+    resized_img = image.resize((target_width, target_height))
+    processed_images = []
+    for i in range(blocks):
+        box = (
+            (i % (target_width // image_size)) * image_size,
+            (i // (target_width // image_size)) * image_size,
+            ((i % (target_width // image_size)) + 1) * image_size,
+            ((i // (target_width // image_size)) + 1) * image_size
+        )
+        # split the image
+        split_img = resized_img.crop(box)
+        processed_images.append(split_img)
+    assert len(processed_images) == blocks
+    if use_thumbnail and len(processed_images) != 1:
+        thumbnail_img = image.resize((image_size, image_size))
+        processed_images.append(thumbnail_img)
+    return processed_images
+
+def split_model(model_path, device):
+
+    device_map = {}
+    world_size = torch.cuda.device_count()
+    config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
+    num_layers = config.llm_config.num_hidden_layers
+
+    num_layers_per_gpu_ = math.floor(num_layers / (world_size - 1))
+    num_layers_per_gpu = [num_layers_per_gpu_] * world_size
+    num_layers_per_gpu[device] = num_layers - num_layers_per_gpu_ * (world_size-1)
+    layer_cnt = 0
+    for i, num_layer in enumerate(num_layers_per_gpu):
+        for j in range(num_layer):
+            device_map[f'language_model.model.layers.{layer_cnt}'] = i
+            layer_cnt += 1
+    device_map['vision_model'] = device
+    device_map['mlp1'] = device
+    device_map['language_model.model.tok_embeddings'] = device
+    device_map['language_model.model.embed_tokens'] = device
+    device_map['language_model.output'] = device
+    device_map['language_model.model.norm'] = device
+    device_map['language_model.lm_head'] = device
+    device_map['language_model.model.rotary_emb'] = device
+    device_map[f'language_model.model.layers.{num_layers - 1}'] = device
+    return device_map
+
+class ModelWorker:
+    def __init__(self, model_path, model_name,
+                 load_8bit, device):
+
+        if model_path.endswith('/'):
+            model_path = model_path[:-1]
+        if model_name is None:
+            model_paths = model_path.split('/')
+            if model_paths[-1].startswith('checkpoint-'):
+                self.model_name = model_paths[-2] + '_' + model_paths[-1]
+            else:
+                self.model_name = model_paths[-1]
+        else:
+            self.model_name = model_name
+
+        print(f'Loading the model {self.model_name}')
+
+        tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, use_fast=False)
+        tokens_to_keep = ['<box>', '</box>', '<ref>', '</ref>']
+        tokenizer.additional_special_tokens = [item for item in tokenizer.additional_special_tokens if item not in tokens_to_keep]
+        self.tokenizer = tokenizer
+        config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
+        model_type = config.vision_config.model_type
+        self.device = torch.cuda.current_device()
+        if model_type == 'siglip_vision_model':
+            self.norm_type = 'siglip'
+        elif model_type == 'MOB':
+            self.norm_type = 'siglip'
+        else:
+            self.norm_type = 'imagenet'
+        print('norm_type: ', self.norm_type)
+        if any(x in model_path.lower() for x in ['34b']):
+            device_map = split_model(model_path, self.device)
+        else:
+            device_map = None
+
+        if device_map is not None:    
+            self.model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,
+                                               low_cpu_mem_usage=True,
+                                               device_map=device_map, 
+                                               trust_remote_code=True,
+                                               load_in_8bit=load_8bit).eval()
+        else:
+            self.model = AutoModel.from_pretrained(model_path, torch_dtype=torch.bfloat16,
+                                               trust_remote_code=True,
+                                               load_in_8bit=load_8bit).eval()  
+        if not load_8bit and device_map is None:
+            self.model = self.model.to(device)
+        self.load_8bit = load_8bit
+        
+        self.model_path = model_path
+        self.image_size = self.model.config.force_image_size
+        self.context_len = tokenizer.model_max_length
+        self.per_tile_len = 256
+        print(self.model)
+    def reload_model(self):
+        del self.model
+        torch.cuda.empty_cache()
+        if self.device == 'auto':
+            os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
+            # This can make distributed deployment work properly
+            self.model = AutoModel.from_pretrained(
+                self.model_path,
+                load_in_8bit=self.load_8bit,
+                torch_dtype=torch.bfloat16,
+                device_map=self.device_map,
+                trust_remote_code=True).eval()
+        else:
+            self.model = AutoModel.from_pretrained(
+                self.model_path,
+                load_in_8bit=self.load_8bit,
+                torch_dtype=torch.bfloat16,
+                trust_remote_code=True).eval()
+        if not self.load_8bit and not self.device == 'auto':
+            self.model = self.model.cuda()
+
+    @torch.inference_mode()
+    def generate(self, params):
+        system_message = params['prompt'][0]['content']
+        send_messages = params['prompt'][1:]
+        max_input_tiles = params['max_input_tiles']
+        temperature = params['temperature']
+        top_p = params['top_p']
+        max_new_tokens = params['max_new_tokens']
+        repetition_penalty = params['repetition_penalty']
+        video_frame_num = params.get('video_frame_num', 64)
+        do_sample = True if temperature > 0.0 else False
+
+        global_image_cnt = 0
+        history, pil_images, max_input_tile_list = [], [], []
+        
+        for message in send_messages:
+            if message['role'] == 'user':
+                prefix = ''
+                if 'image' in message:
+                    for image_data in message['image']:
+                        pil_images.append(load_image(image_data))
+                        prefix = prefix + f'<image {global_image_cnt + 1}><image>\n'
+                        global_image_cnt += 1
+                        max_input_tile_list.append(max_input_tiles)
+                if 'video' in message:
+                    for video_data in message['video']:
+                        video_frames, tmp_prefix = load_video(video_data, num_frames=video_frame_num)
+                        pil_images.extend(video_frames)
+                        prefix = prefix + tmp_prefix
+                        global_image_cnt += len(video_frames)
+                        max_input_tile_list.extend([1] * len(video_frames))
+                content = prefix + message['content']
+                history.append([content, ])
+            else:
+                history[-1].append(message['content'])
+        question, history = history[-1][0], history[:-1]
+
+        if global_image_cnt == 1:
+            question = question.replace('<image 1><image>\n', '<image>\n')
+            history = [[item[0].replace('<image 1><image>\n', '<image>\n'), item[1]] for item in history]
+
+
+        try:
+            assert len(max_input_tile_list) == len(pil_images), 'The number of max_input_tile_list and pil_images should be the same.'
+        except Exception as e:
+            from IPython import embed; embed()
+            exit()
+            print(f'Error: {e}')
+            print(f'max_input_tile_list: {max_input_tile_list}, pil_images: {pil_images}')
+            # raise e
+
+        old_system_message = self.model.system_message
+        self.model.system_message = system_message
+        
+        transform = build_transform(input_size=self.image_size, norm_type=self.norm_type)
+        if len(pil_images) > 0:
+            max_input_tiles_limited_by_contect = params['max_input_tiles']
+            while True:
+                image_tiles = []
+                num_patches_list = []
+                for current_max_input_tiles, pil_image in zip(max_input_tile_list, pil_images):
+                    if self.model.config.dynamic_image_size:
+                        tiles = dynamic_preprocess(
+                            pil_image, image_size=self.image_size, max_num=min(current_max_input_tiles, max_input_tiles_limited_by_contect),
+                            use_thumbnail=self.model.config.use_thumbnail)
+                    else:
+                        tiles = [pil_image]
+                    num_patches_list.append(len(tiles))
+                    image_tiles += tiles
+                if (len(image_tiles) * self.per_tile_len < self.context_len):
+                    break
+                else:
+                    max_input_tiles_limited_by_contect -= 2
+                
+                if max_input_tiles_limited_by_contect < 1:
+                    break
+                    
+            pixel_values = [transform(item) for item in image_tiles]
+
+            
+            pixel_values = torch.stack(pixel_values).to(self.model.device, dtype=torch.bfloat16)
+            
+        else:
+            pixel_values = None
+
+        generation_config = dict(
+            num_beams=1,
+            max_new_tokens=max_new_tokens,
+            do_sample=do_sample,
+            temperature=temperature,
+            repetition_penalty=repetition_penalty,
+            max_length=self.context_len,
+            top_p=top_p,
+        )
+        print(f'pixel_values: {pixel_values.shape}')
+        response = self.model.chat(
+            tokenizer=self.tokenizer,
+            pixel_values=pixel_values,
+            question=question,
+            history=history,
+            return_history=False,
+            num_patches_list=num_patches_list,
+            generation_config=generation_config,
+        )
+        self.model.system_message = old_system_message
+        return {'text': response, 'error_code': 0}
+
+
+
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model-path', type=str, default='/home/zhidingy/workspace/eagle-next/internvl_chat/work_dirs/release/test/Eagle2-2B')
+    parser.add_argument('--model-name', type=str, default='Eagle2')
+    parser.add_argument('--device', type=str, default='cuda')
+    parser.add_argument('--load-8bit', action='store_true')
+    args = parser.parse_args()
+    print(f'args: {args}')
+
+    worker = ModelWorker(
+                         args.model_path,
+                         args.model_name,
+                         args.load_8bit,
+                         args.device)
+    prompt = [
+        {'role': 'system', 'content': 'You are a helpful assistant.'},
+        {'role': 'user', 'content': 'Describe these two images in details respectively.', 
+            'image':[
+                {'url': 'https://www.nvidia.com/content/dam/en-zz/Solutions/about-nvidia/logo-and-brand/[email protected]'},
+                {'url': "https://www.google.com.hk/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png"}
+            ]
+        }
+    ]
+    params = {
+        'prompt': prompt,
+        'max_input_tiles': 24,
+        'temperature': 0.7,
+        'top_p': 1.0,
+        'max_new_tokens': 4096,
+        'repetition_penalty': 1.0,
+    }
+    print(worker.generate(params))

Eagle2-2B/generation_config.json

@@ -0,0 +1,4 @@
+{
+  "_from_model_config": true,
+  "transformers_version": "4.37.2"
+}

Eagle2-2B/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f73a9fc7fb024f2405af6970c5945c3d3bfe160757119acf8ef335f1f10a4235
+size 4428702200

Eagle2-2B/modeling_eagle_chat.py

@@ -0,0 +1,450 @@
+# --------------------------------------------------------
+# Eagle2
+# Copyright (c) 2025 NVIDIA
+# Licensed under The Apache License [see LICENSE for details]
+# --------------------------------------------------------
+
+import warnings
+from typing import Any, List, Optional, Tuple, Union
+
+import torch.utils.checkpoint
+import transformers
+from torch import nn
+from torch.nn import CrossEntropyLoss
+from transformers import (AutoModel, GenerationConfig,
+                          LlamaTokenizer, LlamaForCausalLM)
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from transformers.modeling_utils import PreTrainedModel
+from transformers.utils import ModelOutput, logging
+from peft import LoraConfig, get_peft_model
+from transformers.models.siglip.modeling_siglip import SiglipVisionModel
+
+from transformers.models.qwen2.modeling_qwen2 import Qwen2ForCausalLM
+import functools
+
+
+logger = logging.get_logger(__name__)
+from .configuration_eagle_chat import Eagle2ChatConfig
+
+def cleanup_xlora_pre_hooks(model, verbose=True):
+    cleaned = 0
+    for _, m in model.named_modules():
+        d = getattr(m, "_forward_pre_hooks", None)
+        if not isinstance(d, dict):
+            continue
+        for k, cb in list(d.items()):
+            is_xlora = isinstance(cb, functools.partial) and getattr(cb.func, "__name__", "") == "scalings_injection_hook"
+            if is_xlora:
+                try:
+                    d.pop(k, None)
+                    cleaned += 1
+                except Exception:
+                    pass
+    if verbose and cleaned:
+        print(f"[XLORA] cleaned {cleaned} stale pre_hooks")
+
+def version_cmp(v1, v2, op='eq'):
+    import operator
+
+    from packaging import version
+    op_func = getattr(operator, op)
+    return op_func(version.parse(v1), version.parse(v2))
+
+
+class Eagle2ChatModel(PreTrainedModel):
+    config_class = Eagle2ChatConfig
+    main_input_name = 'pixel_values'
+    _no_split_modules = ['LlamaDecoderLayer']
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+    _supports_flex_attn = False
+    _supports_cache_class = False
+    _supports_quantized_cache = False
+    _supports_static_cache = False
+    _supports_attention_backend = False
+    
+    def __init__(self, config: Eagle2ChatConfig, vision_model=None, language_model=None):
+        super().__init__(config)
+
+        image_size = config.force_image_size or config.vision_config.image_size
+
+        patch_size = config.vision_config.patch_size
+        self.patch_size = patch_size
+        self.num_image_token = int((image_size // patch_size) ** 2 * (config.downsample_ratio ** 2))
+
+        self.select_layer = config.select_layer
+        self.template = config.template
+        self.downsample_ratio = config.downsample_ratio
+
+        logger.info(f'num_image_token: {self.num_image_token}')
+        if vision_model is not None:
+            self.vision_model = vision_model
+        else:
+            if config.vision_config.model_type == 'siglip_vision_model':
+                if version_cmp(transformers.__version__, '4.43.0', 'le'):
+                    config.vision_config._attn_implementation = 'eager'
+                self.vision_model = SiglipVisionModel(config.vision_config)
+
+        if language_model is not None:
+            self.language_model = language_model
+        else:
+            if config.llm_config.architectures[0] == 'LlamaForCausalLM':
+                self.language_model = LlamaForCausalLM(config.llm_config)
+            elif config.llm_config.architectures[0] == 'Qwen2ForCausalLM':
+                self.language_model = Qwen2ForCausalLM(config.llm_config)
+            else:
+                raise NotImplementedError(f'{config.llm_config.architectures[0]} is not implemented.')
+
+        vit_hidden_size = config.vision_config.hidden_size
+
+        llm_hidden_size = config.llm_config.hidden_size
+
+        self.mlp1 = nn.Sequential(
+                nn.LayerNorm(vit_hidden_size * int(1 / self.downsample_ratio) ** 2),
+                nn.Linear(vit_hidden_size * int(1 / self.downsample_ratio) ** 2, llm_hidden_size),
+                nn.GELU(),
+                nn.Linear(llm_hidden_size, llm_hidden_size)
+            )
+        self.img_context_token_id = None
+        self.system_message = 'You are a helpful assistant.' # Default system message
+
+        if config.use_backbone_lora:
+            self.wrap_backbone_lora(r=config.use_backbone_lora, lora_alpha=2 * config.use_backbone_lora)
+
+        if config.use_llm_lora:
+            self.wrap_llm_lora(r=config.use_llm_lora, lora_alpha=2 * config.use_llm_lora)
+    
+    def wrap_backbone_lora(self, r=128, lora_alpha=256, lora_dropout=0.05):
+        lora_config = LoraConfig(
+            r=r,
+            target_modules=['attn.qkv', 'attn.proj', 'mlp.fc1', 'mlp.fc2'],
+            lora_alpha=lora_alpha,
+            lora_dropout=lora_dropout,
+        )
+        self.vision_model = get_peft_model(self.vision_model, lora_config)
+        self.vision_model.print_trainable_parameters()
+
+    def wrap_llm_lora(self, r=128, lora_alpha=256, lora_dropout=0.05):    
+        lora_config = LoraConfig(
+            r=r,
+            target_modules=['self_attn.q_proj', 'self_attn.k_proj', 'self_attn.v_proj', 'self_attn.o_proj',
+                            'mlp.gate_proj', 'mlp.down_proj', 'mlp.up_proj'],
+            lora_alpha=lora_alpha,
+            lora_dropout=lora_dropout,
+            task_type='CAUSAL_LM'
+        )
+        self.language_model = get_peft_model(self.language_model, lora_config)
+        self.language_model.enable_input_require_grads()
+        self.language_model.print_trainable_parameters()
+
+
+    def forward(
+            self,
+            pixel_values: torch.FloatTensor,
+            input_ids: torch.LongTensor = None,
+            attention_mask: Optional[torch.Tensor] = None,
+            position_ids: Optional[torch.LongTensor] = None,
+            image_flags: Optional[torch.LongTensor] = None,
+            past_key_values: Optional[List[torch.FloatTensor]] = None,
+            labels: Optional[torch.LongTensor] = None,
+            use_cache: Optional[bool] = None,
+            output_attentions: Optional[bool] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+            num_patches_list: Optional[List[torch.Tensor]] = None,
+            **kwargs
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        image_flags = image_flags.squeeze(-1)
+        input_embeds = self.language_model.get_input_embeddings()(input_ids)
+
+        vit_embeds = self.extract_feature(pixel_values)
+
+        if not isinstance(image_flags, list):
+            image_flags = image_flags.squeeze(-1)
+            vit_embeds = vit_embeds[image_flags == 1]
+
+        vit_batch_size = pixel_values.shape[0]
+
+        B, N, C = input_embeds.shape
+        input_embeds = input_embeds.reshape(B * N, C)
+
+        # if torch.distributed.get_rank() == 0:
+        #     print(f'dynamic ViT batch size: {vit_batch_size}, images per sample: {vit_batch_size / B}, dynamic token length: {N}')
+
+        input_ids = input_ids.reshape(B * N)
+        selected = (input_ids == self.img_context_token_id)
+        try:
+            input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds.reshape(-1, C)
+        except Exception as e:
+            vit_embeds = vit_embeds.reshape(-1, C)
+            if torch.distributed.get_rank() == 0:
+                print(pixel_values.shape,input_embeds.shape)
+                print(f'warning: {e}, input_embeds[selected].shape={input_embeds[selected].shape}, '
+                    f'vit_embeds.shape={vit_embeds.shape}')
+            n_token = selected.sum()
+            input_embeds[selected] = input_embeds[selected] * 0.0 + vit_embeds[:n_token]
+
+        input_embeds = input_embeds.reshape(B, N, C)
+
+        outputs = self.language_model(
+            inputs_embeds=input_embeds,
+            labels=labels,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            **kwargs
+        )
+        logits = outputs.logits
+
+        loss = None
+        if labels is not None and outputs.loss is None:
+            # Shift so that tokens < n predict n
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.language_model.config.vocab_size)
+            shift_labels = shift_labels.view(-1)
+            # Enable model parallelism
+            shift_labels = shift_labels.to(shift_logits.device)
+            loss = loss_fct(shift_logits, shift_labels)
+        else:
+            loss = outputs.loss
+
+        if not return_dict:
+            output = (logits,) + outputs[1:]
+            return (loss,) + output if loss is not None else output
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def pixel_shuffle(self, x, scale_factor=0.5):
+        n, w, h, c = x.size()
+        # N, W, H, C --> N, W, H * scale, C // scale
+        x = x.view(n, w, int(h * scale_factor), int(c / scale_factor))
+        # N, W, H * scale, C // scale --> N, H * scale, W, C // scale
+        x = x.permute(0, 2, 1, 3).contiguous()
+        # N, H * scale, W, C // scale --> N, H * scale, W * scale, C // (scale ** 2)
+        x = x.view(n, int(h * scale_factor), int(w * scale_factor),
+                   int(c / (scale_factor * scale_factor)))
+        x = x.permute(0, 2, 1, 3).contiguous()
+        return x
+
+    def extract_feature(self, pixel_values):
+        """
+        """
+        
+        if self.select_layer == -1:
+            vit_embeds = self.vision_model(
+                pixel_values=pixel_values,
+                output_hidden_states=False,
+                return_dict=True)
+            # if there is vit_embeds.last_hidden_state, use it.
+            if hasattr(vit_embeds, 'last_hidden_state'):
+                vit_embeds = vit_embeds.last_hidden_state
+        else:
+            vit_embeds = self.vision_model(
+                pixel_values=pixel_values,
+                output_hidden_states=True,
+                return_dict=True).hidden_states[self.select_layer]
+        if type(self.vision_model) == SiglipVisionModel:
+            pass
+        else:
+            vit_embeds = vit_embeds[:, 1:, :] # torch.Size([B, 1024, 1024])
+
+        if self.training and self.neftune_alpha is not None:
+            vit_embeds = self.noised_embed(vit_embeds, self.neftune_alpha)
+
+
+        h = w = int(vit_embeds.shape[1] ** 0.5)
+        vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], h, w, -1)
+
+        vit_embeds = self.pixel_shuffle(vit_embeds, scale_factor=self.downsample_ratio) # torch.Size([B, 1024, 1024]) -> torch.Size([B, 16, 16, 4096])
+        vit_embeds = vit_embeds.reshape(vit_embeds.shape[0], -1, vit_embeds.shape[-1]) # torch.Size([B, 16, 16, 4096]) -> torch.Size([B, 256, 4096])
+        vit_embeds = self.mlp1(vit_embeds)#.to(pixel_values.device)
+
+        return vit_embeds
+    
+    def batch_chat(self, 
+                   tokenizer, pixel_values, questions, generation_config,     num_patches_list=None,
+                   history=None, return_history=False, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>',
+                   IMG_CONTEXT_TOKEN='<IMG_CONTEXT>', verbose=False, image_counts=None):
+        if history is not None or return_history:
+            print('Now multi-turn chat is not supported in batch_chat.')
+            raise NotImplementedError
+
+        if image_counts is not None:
+            num_patches_list = image_counts
+            print('Warning: `image_counts` is deprecated. Please use `num_patches_list` instead.')
+
+        img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
+        self.img_context_token_id = img_context_token_id
+
+        if verbose and pixel_values is not None:
+            image_bs = pixel_values.shape[0]
+            print(f'dynamic ViT batch size: {image_bs}')
+
+        queries = []
+        for idx, num_patches in enumerate(num_patches_list):
+            question = questions[idx]
+            if pixel_values is not None and '<image>' not in question:
+                question = '<image>\n' + question
+            template_messages = []
+            sep = tokenizer.eos_token
+            template_messages.append(('<|im_start|>user', question))
+            template_messages.append(('<|im_end|>assistant', None))
+            query = self.get_prompt(self.system_message, template_messages, sep)
+
+            image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * self.num_image_token * num_patches + IMG_END_TOKEN
+            query = query.replace('<image>', image_tokens, 1)
+            queries.append(query)
+
+        tokenizer.padding_side = 'left'
+        model_inputs = tokenizer(queries, return_tensors='pt', padding=True)
+        input_ids = model_inputs['input_ids'].cuda()
+        attention_mask = model_inputs['attention_mask'].cuda()
+        eos_token_id = tokenizer.convert_tokens_to_ids(sep)
+        generation_config['eos_token_id'] = eos_token_id
+        generation_output = self.generate(
+            pixel_values=pixel_values,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            **generation_config
+        )
+        responses = tokenizer.batch_decode(generation_output, skip_special_tokens=True)
+        responses = [response.split(sep)[0].strip() for response in responses]
+        return responses
+
+    def chat(self, tokenizer, pixel_values, question, generation_config, history=None, return_history=False,
+             num_patches_list=None, IMG_START_TOKEN='<img>', IMG_END_TOKEN='</img>', IMG_CONTEXT_TOKEN='<IMG_CONTEXT>',
+             verbose=False, llm_only=False):
+
+        if history is None and pixel_values is not None and '<image>' not in question:
+            question = '<image>\n' + question
+
+        if num_patches_list is None:
+            num_patches_list = [pixel_values.shape[0]] if pixel_values is not None else []
+        assert pixel_values is None or len(pixel_values) == sum(num_patches_list)
+
+        img_context_token_id = tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
+        self.img_context_token_id = img_context_token_id
+
+        template_messages = []
+        system_message = f'<|im_start|>system\n{self.system_message}'
+        sep = tokenizer.eos_token
+        eos_token_id = tokenizer.convert_tokens_to_ids(sep)
+
+        history = [] if history is None else history
+        for (old_question, old_answer) in history:
+            template_messages.append(('<|im_start|>user', old_question))
+            template_messages.append(('<|im_start|>assistant', old_answer))
+        template_messages.append(('<|im_start|>user', question))
+        template_messages.append(('<|im_end|>assistant', None))
+        query = self.get_prompt(system_message, template_messages, sep)
+
+        if verbose and pixel_values is not None:
+            image_bs = pixel_values.shape[0]
+            print(f'dynamic ViT batch size: {image_bs}')
+
+        for num_patches in num_patches_list:
+            image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * self.num_image_token * num_patches + IMG_END_TOKEN
+            if llm_only:
+                query = query.replace('<image>', '', 1)
+            else:
+                query = query.replace('<image>', image_tokens, 1)
+        
+        model_inputs = tokenizer(query, return_tensors='pt')
+        input_ids = model_inputs['input_ids'].cuda()
+        attention_mask = model_inputs['attention_mask'].cuda()
+        generation_config['eos_token_id'] = eos_token_id
+        generation_output = self.generate(
+            pixel_values=pixel_values,
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            **generation_config
+        )
+        response = tokenizer.batch_decode(generation_output, skip_special_tokens=True)[0]
+        response = response.split(sep)[0].strip()
+        history.append((question, response))
+        if return_history:
+            return response, history
+        else:
+            query_to_print = query.replace(IMG_CONTEXT_TOKEN, '')
+            query_to_print = query_to_print.replace(f'{IMG_START_TOKEN}{IMG_END_TOKEN}', '<image>')
+            if verbose:
+                print(query_to_print, response)
+            return response
+
+    def get_prompt(self, system_prompt, messages, sep) -> str:
+        """Get the prompt for generation."""
+        
+        ret = '' if system_prompt == '' else system_prompt + sep + '\n'
+        for role, message in messages:
+            if message:
+                ret += role + '\n' + message + sep + '\n'
+            else:
+                ret += role + '\n'
+        return ret
+        
+    @torch.no_grad()
+    def generate(
+            self,
+            pixel_values: Optional[torch.FloatTensor] = None,
+            input_ids: Optional[torch.FloatTensor] = None,
+            attention_mask: Optional[torch.LongTensor] = None,
+            visual_features: Optional[torch.FloatTensor] = None,
+            generation_config: Optional[GenerationConfig] = None,
+            output_hidden_states: Optional[bool] = None,
+            return_dict: Optional[bool] = None,
+            **generate_kwargs,
+    ) -> torch.LongTensor:
+
+        assert self.img_context_token_id is not None
+        if pixel_values is not None:
+            if visual_features is not None:
+                vit_embeds = visual_features
+            else:
+                vit_embeds = self.extract_feature(pixel_values)
+
+            input_embeds = self.language_model.get_input_embeddings()(input_ids)
+            B, N, C = input_embeds.shape
+            input_embeds = input_embeds.reshape(B * N, C)
+
+            input_ids = input_ids.reshape(B * N)
+            selected = (input_ids == self.img_context_token_id)
+            assert selected.sum() != 0
+            input_embeds[selected] = vit_embeds.reshape(-1, C).to(input_embeds.device, dtype=input_embeds.dtype)
+
+            input_embeds = input_embeds.reshape(B, N, C)
+        else:
+            input_embeds = self.language_model.get_input_embeddings()(input_ids)
+
+        outputs = self.language_model.generate(
+            inputs_embeds=input_embeds,
+            attention_mask=attention_mask,
+            generation_config=generation_config,
+            output_hidden_states=output_hidden_states,
+            # return_dict=return_dict, # default is True
+            **generate_kwargs,
+        )
+
+        cleanup_xlora_pre_hooks(self.language_model)
+
+
+        return outputs
+        
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+    
+    def get_output_embeddings(self):
+        return self.language_model.get_output_embeddings()

Eagle2-2B/special_tokens_map.json

@@ -0,0 +1,40 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<img>",
+    "</img>",
+    "<IMG_CONTEXT>",
+    "<quad>",
+    "</quad>",
+    "<ref>",
+    "</ref>",
+    "<box>",
+    "</box>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

Eagle2-2B/tokenizer_config.json

@@ -0,0 +1,289 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<img>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151666": {
+      "content": "</img>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151667": {
+      "content": "<IMG_CONTEXT>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151668": {
+      "content": "<quad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151669": {
+      "content": "</quad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<ref>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "</ref>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<box>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151673": {
+      "content": "</box>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<img>",
+    "</img>",
+    "<IMG_CONTEXT>",
+    "<quad>",
+    "</quad>",
+    "<ref>",
+    "</ref>",
+    "<box>",
+    "</box>"
+  ],
+  "bos_token": null,
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- else %}\n        {{- '<|im_start|>system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 16384,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

empty_language_adapter/README.md

@@ -0,0 +1,202 @@
+---
+base_model: ./pretrained/Qwen2_5-1_5B-Instruct
+library_name: peft
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.15.2

empty_language_adapter/adapter_config.json

@@ -0,0 +1,32 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "./pretrained/Qwen2_5-1_5B-Instruct",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 128,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "self_attn.o_proj",
+    "mlp.up_proj",
+    "self_attn.v_proj",
+    "self_attn.k_proj",
+    "mlp.down_proj",
+    "self_attn.q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

empty_language_adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dada5c48f8e5936e18bda0930a998cecb8fa9e2aea0abea387f5647eae7134a7
+size 440447544