Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,106 @@
----
-license: apache-2.0
----
+### Ops-MM-embedding-v1-2B
+
+**Ops-MM-embedding-v1-2B** is a dense, large-scale multimodal embedding model developed and open-sourced by the Alibaba Cloud OpenSearch-AI team, fine-tuned from Qwen2-VL.
+
+---
+
+### **Key Features**
+
+#### Unified Multimodal Embeddings
+- Encodes text, images, text-image pairs, visual documents, and videos (by treating video frames as multiple image inputs) into a unified embedding space for cross-modal retrieval.
+
+#### High Performance on MMEB
+- Achieves **SOTA results** among models of similar scale on **MMEB-V2** and **MMEB-Image** benchmark (until 2025-07-03).
+
+#### Multilingual Capabilities
+- The larger variant (**Ops-MM-embedding-v1-7B**) achieves SOTA performance among dense models on the ViDoRe-v2 benchmark, demonstrating strong cross-lingual generalization.
+
+
+
+### Training data
+
+MMEB-train, CC-3M, colpali training set.
+
+
+### Performance
+
+#### MMEB-V2
+
+| Model                    | Model Size (B) | Overall | Image-Overall | Video-Overall | Visdoc-Overall |
+| ------------------------ | -------------- | ------- | ------------- | ------------- | -------------- |
+| seed-1.6-embedding       | unknown        | 71.57   | 77.78         | 55.34         | 74.41          |
+| Ops-MM-embedding-v1-7B   | 8.29           | 67.79   | 72.72         | 53.76         | 70.91          |
+| Ops-MM-embedding-v1-2B   | 2.21           | 63.62   | 69.03         | 47.56         | 67.55          |
+| VLM2Vec-V2.0-Qwen2VL-2B  | 2.21           | 58.39   | 64.85         | 34.85         | 66.34          |
+| gme-Qwen2-VL-2B-Instruct | 2.21           | 54.37   | 51.89         | 33.86         | 73.47          |
+
+---
+
+#### MMEB-Image
+
+The table below compares performance on MMEB-Image benchmark among models of similar size.
+
+| Model                  | Model Size (B) | Image-Overall | I-CLS | I-QA  | I-RET | I-VG  |
+| ---------------------- | -------------- | ------------- | ----- | ----- | ----- | ----- |
+| Ops-MM-embedding-v1-2B | 2.21           | **69.03**     | 68.07 | 65.11 | 69.17 | 80.85 |
+| B3_Qwen2_2B            | 2.21           | 68.1          | 67    | 61.19 | 70.85 | 79.88 |
+| LLaVE-2B               | 1.95           | 65.2          | 62.1  | 60.2  | 65.2  | 84.9  |
+
+---
+
+#### ViDoRe-v2
+
+| Model                  | Avg      | ESG Restaurant Human | MIT Bio | Econ. Macro | ESG Restaurant Synth. | MIT Bio Multi. | Econ Macro Multi. | ESG Restaurant Synth. Multi. |
+| ---------------------- | -------- | -------------------- | ------- | ----------- | --------------------- | -------------- | ----------------- | ---------------------------- |
+| gme-7B                 | 59.3     | 65.8                 | 64      | 62.9        | 54.3                  | 55.1           | 56.2              | 56.7                         |
+| seed 1.6 embedding     | 58.9     | 63.3                 | 63.9    | 64.0        | 58.4                  | 57.1           | 53.8              | 52.0                         |
+| Ops-MM-embedding-v1-7B | **60.6** | 66.3                 | 58.4    | 67.4        | 60.0                  | 54.3           | 60.9              | 56.8                         |
+| Ops-MM-embedding-v1-2B | 54.4     | 58.6                 | 56.0    | 56.4        | 55.8                  | 52.9           | 47.9              | 53.4                         |
+
+
+
+
+## Usage
+
+```python
+from ops_mm_embedding_v1 import OpsMMEmbeddingV1, fetch_image
+
+
+model = OpsMMEmbeddingV1(
+    "OpenSearch-AI/Ops-MM-embedding-v1-2B",
+    device="cuda",
+    attn_implementation="flash_attention_2"
+)
+
+t2i_prompt = "Find an image that matches the given text."
+texts = [
+    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
+    "Alibaba office.",
+    "Alibaba office.",
+]
+images = [
+    "https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Alibaba_Binjiang_Park.jpg/1024px-Alibaba_Binjiang_Park.jpg"
+]
+
+images = [fetch_image(image) for image in images]
+
+# Text and image embedding
+text_embeddings = model.get_text_embeddings(texts)
+image_embeddings = model.get_image_embeddings(images)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Fused Embedding
+text_with_image_embeddings = model.get_fused_embeddings(texts=texts, images=images, instruction=t2i_prompt)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Multi-image embeddings
+multi_images = [
+    [images[0]],
+    [images[1], images[2]],
+]
+multi_image_embeddings = model.get_image_embeddings(multi_images)
+print('Multi-image embeddings', (multi_image_embeddings @ multi_image_embeddings.T).tolist())
+
+```

added_tokens.json

@@ -0,0 +1,16 @@
+{
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}

config.json

@@ -0,0 +1,56 @@
+{
+  "architectures": [
+    "Qwen2VLForConditionalGeneration"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "image_token_id": 151655,
+  "initializer_range": 0.02,
+  "intermediate_size": 8960,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen2_vl",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 2,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "mrope_section": [
+      16,
+      24,
+      24
+    ],
+    "rope_type": "default",
+    "type": "default"
+  },
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.1",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "video_token_id": 151656,
+  "vision_config": {
+    "depth": 32,
+    "embed_dim": 1280,
+    "hidden_act": "quick_gelu",
+    "hidden_size": 1536,
+    "in_channels": 3,
+    "in_chans": 3,
+    "mlp_ratio": 4,
+    "model_type": "qwen2_vl",
+    "num_heads": 16,
+    "patch_size": 14,
+    "spatial_merge_size": 2,
+    "spatial_patch_size": 14,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652,
+  "vision_token_id": 151654,
+  "vocab_size": 151936
+}

demo.py

@@ -0,0 +1,39 @@
+from ops_mm_embedding_v1 import OpsMMEmbeddingV1, fetch_image
+
+
+model = OpsMMEmbeddingV1(
+    "OpenSearch-AI/Ops-MM-embedding-v1-2B",
+    device="cuda",
+    attn_implementation="flash_attention_2"
+)
+
+t2i_prompt = "Find an image that matches the given text."
+texts = [
+    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
+    "Alibaba office.",
+    "Alibaba office.",
+]
+images = [
+    "https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Alibaba_Binjiang_Park.jpg/1024px-Alibaba_Binjiang_Park.jpg"
+]
+
+images = [fetch_image(image) for image in images]
+
+# Text and image embedding
+text_embeddings = model.get_text_embeddings(texts)
+image_embeddings = model.get_image_embeddings(images)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Fused Embedding
+text_with_image_embeddings = model.get_fused_embeddings(texts=texts, images=images, instruction=t2i_prompt)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Multi-image embeddings
+multi_images = [
+    [images[0]],
+    [images[1], images[2]],
+]
+multi_image_embeddings = model.get_image_embeddings(multi_images)
+print('Multi-image embeddings', (multi_image_embeddings @ multi_image_embeddings.T).tolist())

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.01,
+  "top_k": 1,
+  "top_p": 0.001,
+  "transformers_version": "4.51.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a94699edc3bfe4d03f7cd060b5fe9b153cbe45e810cf09ef52a15d2ea92a61f
+size 4418050848

ops_mm_embedding_v1.py

@@ -0,0 +1,309 @@
+import math
+from typing import List, Optional, TypeAlias, Union
+
+import torch
+import torch.nn as nn
+from PIL import Image
+from tqdm import tqdm
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+ImageInput: TypeAlias = Union[Image.Image, List[Image.Image]]
+BatchImageInput: TypeAlias = Union[List[Image.Image], List[List[Image.Image]]]
+
+
+class OpsMMEmbeddingV1(nn.Module):
+    def __init__(
+        self,
+        model_name: str,
+        device: str = "cuda",
+        max_length: Optional[int] = None,
+        attn_implementation: Optional[str] = None,
+    ):
+        super().__init__()
+        self.device = device
+        self.max_length = max_length
+        self.default_instruction = "You are a helpful assistant."
+        self.base_model = AutoModelForImageTextToText.from_pretrained(
+            model_name,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+            attn_implementation=attn_implementation,
+        ).to(self.device)
+
+        self.processor = AutoProcessor.from_pretrained(model_name, min_pixels=256 * 28 * 28, max_pixels=1280 * 28 * 28)
+        self.processor.tokenizer.padding_side = "left"
+        self.eval()
+
+    def encode_input(self, input):
+        hidden_states = self.base_model(**input, return_dict=True, output_hidden_states=True)
+        hidden_states = hidden_states.hidden_states[-1]
+        pooled_output = self._pooling(hidden_states)
+        return pooled_output
+
+    def _pooling(self, last_hidden_state):
+        batch_size = last_hidden_state.shape[0]
+        reps = last_hidden_state[torch.arange(batch_size), -1, :]
+        reps = torch.nn.functional.normalize(reps, p=2, dim=-1)
+        return reps
+
+    def _validate_instructions(
+        self,
+        texts: Optional[List[str]],
+        images: Optional[BatchImageInput],
+        instruction: Optional[Union[str, List[str]]],
+    ) -> List[str]:
+        """Validate and format instructions to match batch size"""
+        batch_size = max(len(x) if x is not None else 0 for x in [texts, images])
+
+        if instruction is None:
+            return [self.default_instruction] * batch_size
+
+        if isinstance(instruction, str):
+            return [instruction] * batch_size
+
+        if isinstance(instruction, list):
+            if len(instruction) != batch_size:
+                raise ValueError(f"Length of instruction list ({len(instruction)}) must match batch size ({batch_size}) when texts/images are provided")
+            return instruction
+
+        raise TypeError("instruction must be str, List[str] or None")
+
+    def _process_images(self, images: ImageInput) -> List[Image.Image]:
+        """Convert single image or list of images to processed format"""
+        if isinstance(images, Image.Image) or isinstance(images, str):
+            return [fetch_image(images)]
+        return [fetch_image(i) for i in images]
+
+    def embed(
+        self,
+        texts: Optional[List[str]] = None,
+        images: Optional[BatchImageInput] = None,
+        instruction: Optional[Union[str, List[str]]] = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Generate embeddings for text, images, or combined inputs.
+
+        Args:
+            texts: List of text inputs (optional)
+            images: Can be:
+                - List[Image.Image]: Single image per input
+                - List[List[Image.Image]]: Multiple images per input
+            instruction: Instruction(s) for the model. Can be:
+                - None: use default instruction
+                - str: use same instruction for all inputs
+                - List[str]: per-input instructions (must match batch size)
+        """
+        if texts is None and images is None:
+            raise ValueError("Either texts or images must be provided")
+
+        instructions = self._validate_instructions(texts, images, instruction)
+
+        # Determine batch size
+        batch_size = len(texts) if texts is not None else len(images)  # type: ignore
+
+        input_texts, input_images = [], []
+        for i in range(batch_size):
+            text = texts[i] if texts is not None else None
+            image = images[i] if images is not None else None
+
+            input_str = ""
+            processed_image = None
+            if image is not None:
+                processed_image = self._process_images(image)
+                input_str += "<|vision_start|><|image_pad|><|vision_end|>" * len(processed_image)
+
+            if text is not None:
+                input_str += text
+
+            msg = f"<|im_start|>system\n{instructions[i]}<|im_end|>\n<|im_start|>user\n{input_str}<|im_end|>\n<|im_start|>assistant\n<|endoftext|>"
+
+            input_texts.append(msg)
+            input_images.append(processed_image)
+
+        # Only pass to processor if we actually have images
+        processed_images = input_images if any(img is not None for img in input_images) else None
+
+        inputs = self.processor(
+            text=input_texts,
+            images=processed_images,
+            padding=True,
+            truncation=True,
+            max_length=self.max_length,
+            return_tensors="pt",
+        )
+        inputs = {k: v.to(self.device) for k, v in inputs.items()}
+
+        with torch.inference_mode():
+            embeddings = self.encode_input(inputs)
+
+        return embeddings
+
+    def get_text_embeddings(
+        self,
+        texts: List[str],
+        instruction: Optional[Union[str, List[str]]] = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Convenience method for text-only embeddings"""
+        return self.get_fused_embeddings(texts=texts, instruction=instruction, **kwargs)
+
+    def get_image_embeddings(
+        self,
+        images: BatchImageInput,
+        instruction: Optional[Union[str, List[str]]] = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Convenience method for image-only embeddings.
+
+        Args:
+            images: Can be:
+                - List[Image.Image]: Single image per input
+                - List[List[Image.Image]]: Multiple images per input
+        """
+        return self.get_fused_embeddings(images=images, instruction=instruction, **kwargs)
+
+    def get_fused_embeddings(
+        self,
+        texts: Optional[List[str]] = None,
+        images: Optional[BatchImageInput] = None,
+        instruction: Optional[Union[str, List[str]]] = None,
+        batch_size: int = 8,
+        show_progress: bool = True,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Batch processing for large collections of texts/images.
+
+        Args:
+            texts: List of text inputs (optional)
+            images: Can be:
+                - List[Image.Image]: Single image per input
+                - List[List[Image.Image]]: Multiple images per input
+            instruction: Instruction(s) for the model
+            batch_size: Number of items to process at once
+            show_progress: Whether to display progress bar
+        """
+
+        if texts is None and images is None:
+            raise ValueError("Either texts or images must be provided")
+
+        total_items = len(texts) if texts is not None else len(images)  # type: ignore
+        num_batches = math.ceil(total_items / batch_size)
+
+        all_embeddings = []
+        progress = tqdm(total=num_batches, disable=not show_progress, desc="Processing")
+
+        for i in range(0, total_items, batch_size):
+            batch_texts = texts[i : i + batch_size] if texts is not None else None
+            batch_images = images[i : i + batch_size] if images is not None else None
+            batch_emb = self.embed(texts=batch_texts, images=batch_images, instruction=instruction)
+
+            all_embeddings.append(batch_emb.cpu())
+            progress.update(1)
+
+        progress.close()
+        return torch.cat(all_embeddings, dim=0).to(self.device)
+
+    def forward(self, **inputs) -> torch.Tensor:
+        """Alias for encode_input"""
+        return self.encode_input(inputs)
+
+
+### Modified from qwen_vl_utils.vision_process.py
+import base64
+import logging
+import math
+from io import BytesIO
+
+import requests
+
+IMAGE_FACTOR = 28
+MIN_PIXELS = 256 * 28 * 28
+MAX_PIXELS = 1280 * 28 * 28
+MAX_RATIO = 200
+
+
+def round_by_factor(number: int, factor: int) -> int:
+    """Returns the closest integer to 'number' that is divisible by 'factor'."""
+    return round(number / factor) * factor
+
+
+def ceil_by_factor(number: int | float, factor: int) -> int:
+    """Returns the smallest integer greater than or equal to 'number' that is divisible by 'factor'."""
+    return math.ceil(number / factor) * factor
+
+
+def floor_by_factor(number: int | float, factor: int) -> int:
+    """Returns the largest integer less than or equal to 'number' that is divisible by 'factor'."""
+    return math.floor(number / factor) * factor
+
+
+def smart_resize(
+    height: int,
+    width: int,
+    factor: int = IMAGE_FACTOR,
+    min_pixels: int = MIN_PIXELS,
+    max_pixels: int = MAX_PIXELS,
+) -> tuple[int, int]:
+    """
+    Rescales the image so that the following conditions are met:
+    1. Both dimensions (height and width) are divisible by 'factor'.
+    2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].
+    3. The aspect ratio of the image is maintained as closely as possible.
+    """
+    h_bar = max(factor, round_by_factor(height, factor))
+    w_bar = max(factor, round_by_factor(width, factor))
+    if h_bar * w_bar > max_pixels:
+        beta = math.sqrt((height * width) / max_pixels)
+        h_bar = floor_by_factor(height / beta, factor)
+        w_bar = floor_by_factor(width / beta, factor)
+    elif h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (height * width))
+        h_bar = ceil_by_factor(height * beta, factor)
+        w_bar = ceil_by_factor(width * beta, factor)
+
+    if max(h_bar, w_bar) / min(h_bar, w_bar) > MAX_RATIO:
+        logging.warning(f"Absolute aspect ratio must be smaller than {MAX_RATIO}, got {max(h_bar, w_bar) / min(h_bar, w_bar)}")
+        if h_bar > w_bar:
+            h_bar = w_bar * MAX_RATIO
+        else:
+            w_bar = h_bar * MAX_RATIO
+    return h_bar, w_bar
+
+
+def fetch_image(
+    image: str | Image.Image,
+    size_factor: int = IMAGE_FACTOR,
+    min_pixels: int = MIN_PIXELS,
+    max_pixels: int = MAX_PIXELS,
+) -> Image.Image:
+    image_obj = None
+    if isinstance(image, Image.Image):
+        image_obj = image
+    elif image.startswith("http://") or image.startswith("https://"):
+        image_obj = Image.open(requests.get(image, stream=True).raw)  # type: ignore
+    elif image.startswith("file://"):
+        image_obj = Image.open(image[7:])
+    elif image.startswith("data:image"):
+        if "base64," in image:
+            _, base64_data = image.split("base64,", 1)
+            data = base64.b64decode(base64_data)
+            image_obj = Image.open(BytesIO(data))
+    else:
+        image_obj = Image.open(image)
+    if image_obj is None:
+        raise ValueError(f"Unrecognized image input, support local path, http url, base64 and PIL.Image, got {image}")
+    image = image_obj.convert("RGB")
+    width, height = image.size
+    resized_height, resized_width = smart_resize(
+        height,
+        width,
+        factor=size_factor,
+        min_pixels=min_pixels,
+        max_pixels=max_pixels,
+    )
+    image = image.resize((resized_width, resized_height))
+
+    return image
+
+
+###

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 12845056,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "patch_size": 14,
+  "processor_class": "Qwen2VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 12845056,
+    "shortest_edge": 3136
+  },
+  "temporal_patch_size": 2
+}

score/Ops-MM-embedding-v1-2B.json

@@ -0,0 +1,2289 @@
+{
+    "metadata": {
+        "model_name": "Ops-MM-embedding-v1-2B",
+        "model_backbone": "Qwen2-VL-2B",
+        "model_size": 2.21,
+        "embedding_dimension": 1536,
+        "max_length_tokens": null,
+        "model_release_date": "2025-07-03",
+        "data_source": "OpenSearch-AI",
+        "url": "https://huggingface.co/OpenSearch-AI/Ops-MM-embedding-v1-2B",
+        "report_generated_date": "2025-07-02T22:59:36.793912"
+    },
+    "metrics": {
+        "image": {
+            "ImageNet-1K": {
+                "hit@1": 0.811,
+                "hit@5": 0.961,
+                "hit@10": 0.982,
+                "ndcg_linear@1": 0.811,
+                "ndcg_linear@5": 0.8964826904887964,
+                "ndcg_linear@10": 0.9033237496484268,
+                "ndcg_exponential@1": 0.811,
+                "ndcg_exponential@5": 0.8964826904887964,
+                "ndcg_exponential@10": 0.9033237496484268,
+                "precision@1": 0.811,
+                "precision@5": 0.19220000000000004,
+                "precision@10": 0.09820000000000002,
+                "recall@1": 0.811,
+                "recall@5": 0.961,
+                "recall@10": 0.982,
+                "f1@1": 0.811,
+                "f1@5": 0.3203333333333333,
+                "f1@10": 0.17854545454545453,
+                "map@1": 0.811,
+                "map@5": 0.8745,
+                "map@10": 0.8773535714285714,
+                "mrr@1": 0.811,
+                "mrr@5": 0.8745,
+                "mrr@10": 0.8773535714285714,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "N24News": {
+                "hit@1": 0.784,
+                "hit@5": 0.977,
+                "hit@10": 0.994,
+                "ndcg_linear@1": 0.784,
+                "ndcg_linear@5": 0.8930799148461822,
+                "ndcg_linear@10": 0.8985800069685335,
+                "ndcg_exponential@1": 0.784,
+                "ndcg_exponential@5": 0.8930799148461822,
+                "ndcg_exponential@10": 0.8985800069685335,
+                "precision@1": 0.784,
+                "precision@5": 0.19540000000000005,
+                "precision@10": 0.0994,
+                "recall@1": 0.784,
+                "recall@5": 0.977,
+                "recall@10": 0.994,
+                "f1@1": 0.784,
+                "f1@5": 0.3256666666666666,
+                "f1@10": 0.18072727272727268,
+                "map@1": 0.784,
+                "map@5": 0.8645333333333333,
+                "map@10": 0.866809126984127,
+                "mrr@1": 0.784,
+                "mrr@5": 0.8645333333333333,
+                "mrr@10": 0.866809126984127,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "HatefulMemes": {
+                "hit@1": 0.74,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.74,
+                "ndcg_linear@5": 0.904041735928579,
+                "ndcg_linear@10": 0.904041735928579,
+                "ndcg_exponential@1": 0.74,
+                "ndcg_exponential@5": 0.904041735928579,
+                "ndcg_exponential@10": 0.904041735928579,
+                "precision@1": 0.74,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.74,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.74,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.74,
+                "map@5": 0.87,
+                "map@10": 0.87,
+                "mrr@1": 0.74,
+                "mrr@5": 0.87,
+                "mrr@10": 0.87,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VOC2007": {
+                "hit@1": 0.813,
+                "hit@5": 0.971,
+                "hit@10": 0.99,
+                "ndcg_linear@1": 0.813,
+                "ndcg_linear@5": 0.9042203424068372,
+                "ndcg_linear@10": 0.9105359793935004,
+                "ndcg_exponential@1": 0.813,
+                "ndcg_exponential@5": 0.9042203424068372,
+                "ndcg_exponential@10": 0.9105359793935004,
+                "precision@1": 0.813,
+                "precision@5": 0.19420000000000004,
+                "precision@10": 0.09900000000000002,
+                "recall@1": 0.813,
+                "recall@5": 0.971,
+                "recall@10": 0.99,
+                "f1@1": 0.813,
+                "f1@5": 0.3236666666666666,
+                "f1@10": 0.18,
+                "map@1": 0.813,
+                "map@5": 0.8814000000000001,
+                "map@10": 0.8841063492063492,
+                "mrr@1": 0.813,
+                "mrr@5": 0.8814000000000001,
+                "mrr@10": 0.8841063492063492,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "SUN397": {
+                "hit@1": 0.807,
+                "hit@5": 0.98,
+                "hit@10": 0.991,
+                "ndcg_linear@1": 0.807,
+                "ndcg_linear@5": 0.9053304429453105,
+                "ndcg_linear@10": 0.9090085996069652,
+                "ndcg_exponential@1": 0.807,
+                "ndcg_exponential@5": 0.9053304429453105,
+                "ndcg_exponential@10": 0.9090085996069652,
+                "precision@1": 0.807,
+                "precision@5": 0.19600000000000006,
+                "precision@10": 0.09910000000000002,
+                "recall@1": 0.807,
+                "recall@5": 0.98,
+                "recall@10": 0.991,
+                "f1@1": 0.807,
+                "f1@5": 0.3266666666666666,
+                "f1@10": 0.18018181818181817,
+                "map@1": 0.807,
+                "map@5": 0.8799166666666667,
+                "map@10": 0.8815047619047619,
+                "mrr@1": 0.807,
+                "mrr@5": 0.8799166666666667,
+                "mrr@10": 0.8815047619047619,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Place365": {
+                "hit@1": 0.439,
+                "hit@5": 0.738,
+                "hit@10": 0.815,
+                "ndcg_linear@1": 0.439,
+                "ndcg_linear@5": 0.601791445742296,
+                "ndcg_linear@10": 0.6269267599806451,
+                "ndcg_exponential@1": 0.439,
+                "ndcg_exponential@5": 0.601791445742296,
+                "ndcg_exponential@10": 0.6269267599806451,
+                "precision@1": 0.439,
+                "precision@5": 0.14759999999999998,
+                "precision@10": 0.0815,
+                "recall@1": 0.439,
+                "recall@5": 0.738,
+                "recall@10": 0.815,
+                "f1@1": 0.439,
+                "f1@5": 0.24600000000000005,
+                "f1@10": 0.1481818181818182,
+                "map@1": 0.439,
+                "map@5": 0.5561666666666666,
+                "map@10": 0.5666761904761904,
+                "mrr@1": 0.439,
+                "mrr@5": 0.5561666666666666,
+                "mrr@10": 0.5666761904761904,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ImageNet-A": {
+                "hit@1": 0.531,
+                "hit@5": 0.785,
+                "hit@10": 0.846,
+                "ndcg_linear@1": 0.531,
+                "ndcg_linear@5": 0.6706099804071457,
+                "ndcg_linear@10": 0.6904167945995113,
+                "ndcg_exponential@1": 0.531,
+                "ndcg_exponential@5": 0.6706099804071457,
+                "ndcg_exponential@10": 0.6904167945995113,
+                "precision@1": 0.531,
+                "precision@5": 0.157,
+                "precision@10": 0.08460000000000001,
+                "recall@1": 0.531,
+                "recall@5": 0.785,
+                "recall@10": 0.846,
+                "f1@1": 0.531,
+                "f1@5": 0.26166666666666677,
+                "f1@10": 0.15381818181818183,
+                "map@1": 0.531,
+                "map@5": 0.63205,
+                "map@10": 0.6402698412698412,
+                "mrr@1": 0.531,
+                "mrr@5": 0.63205,
+                "mrr@10": 0.6402698412698412,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ImageNet-R": {
+                "hit@1": 0.912,
+                "hit@5": 0.981,
+                "hit@10": 0.99,
+                "ndcg_linear@1": 0.912,
+                "ndcg_linear@5": 0.9506606903944571,
+                "ndcg_linear@10": 0.9535702649792099,
+                "ndcg_exponential@1": 0.912,
+                "ndcg_exponential@5": 0.9506606903944571,
+                "ndcg_exponential@10": 0.9535702649792099,
+                "precision@1": 0.912,
+                "precision@5": 0.19620000000000004,
+                "precision@10": 0.09900000000000002,
+                "recall@1": 0.912,
+                "recall@5": 0.981,
+                "recall@10": 0.99,
+                "f1@1": 0.912,
+                "f1@5": 0.327,
+                "f1@10": 0.18,
+                "map@1": 0.912,
+                "map@5": 0.9403833333333333,
+                "map@10": 0.9415845238095237,
+                "mrr@1": 0.912,
+                "mrr@5": 0.9403833333333333,
+                "mrr@10": 0.9415845238095237,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ObjectNet": {
+                "hit@1": 0.684,
+                "hit@5": 0.874,
+                "hit@10": 0.911,
+                "ndcg_linear@1": 0.684,
+                "ndcg_linear@5": 0.7906497791119159,
+                "ndcg_linear@10": 0.8028127610985015,
+                "ndcg_exponential@1": 0.684,
+                "ndcg_exponential@5": 0.7906497791119159,
+                "ndcg_exponential@10": 0.8028127610985015,
+                "precision@1": 0.684,
+                "precision@5": 0.1748,
+                "precision@10": 0.0911,
+                "recall@1": 0.684,
+                "recall@5": 0.874,
+                "recall@10": 0.911,
+                "f1@1": 0.684,
+                "f1@5": 0.29133333333333333,
+                "f1@10": 0.16563636363636364,
+                "map@1": 0.684,
+                "map@5": 0.7624166666666667,
+                "map@10": 0.7675519841269842,
+                "mrr@1": 0.684,
+                "mrr@5": 0.7624166666666667,
+                "mrr@10": 0.7675519841269842,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Country211": {
+                "hit@1": 0.286,
+                "hit@5": 0.494,
+                "hit@10": 0.6,
+                "ndcg_linear@1": 0.286,
+                "ndcg_linear@5": 0.3961926672546998,
+                "ndcg_linear@10": 0.43102042795581663,
+                "ndcg_exponential@1": 0.286,
+                "ndcg_exponential@5": 0.3961926672546998,
+                "ndcg_exponential@10": 0.43102042795581663,
+                "precision@1": 0.286,
+                "precision@5": 0.0988,
+                "precision@10": 0.06,
+                "recall@1": 0.286,
+                "recall@5": 0.494,
+                "recall@10": 0.6,
+                "f1@1": 0.286,
+                "f1@5": 0.1646666666666667,
+                "f1@10": 0.10909090909090913,
+                "map@1": 0.286,
+                "map@5": 0.3636833333333333,
+                "map@10": 0.37838095238095243,
+                "mrr@1": 0.286,
+                "mrr@5": 0.3636833333333333,
+                "mrr@10": 0.37838095238095243,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "OK-VQA": {
+                "hit@1": 0.641,
+                "hit@5": 0.893,
+                "hit@10": 0.941,
+                "ndcg_linear@1": 0.641,
+                "ndcg_linear@5": 0.7797349998317328,
+                "ndcg_linear@10": 0.7952131287355804,
+                "ndcg_exponential@1": 0.641,
+                "ndcg_exponential@5": 0.7797349998317328,
+                "ndcg_exponential@10": 0.7952131287355804,
+                "precision@1": 0.641,
+                "precision@5": 0.17860000000000004,
+                "precision@10": 0.09410000000000002,
+                "recall@1": 0.641,
+                "recall@5": 0.893,
+                "recall@10": 0.941,
+                "f1@1": 0.641,
+                "f1@5": 0.2976666666666667,
+                "f1@10": 0.17109090909090907,
+                "map@1": 0.641,
+                "map@5": 0.7416,
+                "map@10": 0.7479642857142857,
+                "mrr@1": 0.641,
+                "mrr@5": 0.7416,
+                "mrr@10": 0.7479642857142857,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "A-OKVQA": {
+                "hit@1": 0.54,
+                "hit@5": 0.847,
+                "hit@10": 0.908,
+                "ndcg_linear@1": 0.54,
+                "ndcg_linear@5": 0.7083891854741314,
+                "ndcg_linear@10": 0.7286378657248158,
+                "ndcg_exponential@1": 0.54,
+                "ndcg_exponential@5": 0.7083891854741314,
+                "ndcg_exponential@10": 0.7286378657248158,
+                "precision@1": 0.54,
+                "precision@5": 0.1694,
+                "precision@10": 0.0908,
+                "recall@1": 0.54,
+                "recall@5": 0.847,
+                "recall@10": 0.908,
+                "f1@1": 0.54,
+                "f1@5": 0.2823333333333334,
+                "f1@10": 0.16509090909090907,
+                "map@1": 0.54,
+                "map@5": 0.6618333333333334,
+                "map@10": 0.670495238095238,
+                "mrr@1": 0.54,
+                "mrr@5": 0.6618333333333334,
+                "mrr@10": 0.670495238095238,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "DocVQA": {
+                "hit@1": 0.93,
+                "hit@5": 0.986,
+                "hit@10": 0.991,
+                "ndcg_linear@1": 0.93,
+                "ndcg_linear@5": 0.9623408127289065,
+                "ndcg_linear@10": 0.9640582325003497,
+                "ndcg_exponential@1": 0.93,
+                "ndcg_exponential@5": 0.9623408127289065,
+                "ndcg_exponential@10": 0.9640582325003497,
+                "precision@1": 0.93,
+                "precision@5": 0.19720000000000004,
+                "precision@10": 0.09910000000000001,
+                "recall@1": 0.93,
+                "recall@5": 0.986,
+                "recall@10": 0.991,
+                "f1@1": 0.93,
+                "f1@5": 0.3286666666666666,
+                "f1@10": 0.1801818181818182,
+                "map@1": 0.93,
+                "map@5": 0.9542333333333334,
+                "map@10": 0.9550011904761905,
+                "mrr@1": 0.93,
+                "mrr@5": 0.9542333333333334,
+                "mrr@10": 0.9550011904761905,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "InfographicsVQA": {
+                "hit@1": 0.653,
+                "hit@5": 0.85,
+                "hit@10": 0.904,
+                "ndcg_linear@1": 0.653,
+                "ndcg_linear@5": 0.7628292030923156,
+                "ndcg_linear@10": 0.7804337590556499,
+                "ndcg_exponential@1": 0.653,
+                "ndcg_exponential@5": 0.7628292030923156,
+                "ndcg_exponential@10": 0.7804337590556499,
+                "precision@1": 0.653,
+                "precision@5": 0.17000000000000004,
+                "precision@10": 0.09040000000000001,
+                "recall@1": 0.653,
+                "recall@5": 0.85,
+                "recall@10": 0.904,
+                "f1@1": 0.653,
+                "f1@5": 0.2833333333333334,
+                "f1@10": 0.16436363636363638,
+                "map@1": 0.653,
+                "map@5": 0.7333500000000001,
+                "map@10": 0.7406988095238096,
+                "mrr@1": 0.653,
+                "mrr@5": 0.7333500000000001,
+                "mrr@10": 0.7406988095238096,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ChartQA": {
+                "hit@1": 0.561,
+                "hit@5": 0.737,
+                "hit@10": 0.784,
+                "ndcg_linear@1": 0.561,
+                "ndcg_linear@5": 0.6571987612384224,
+                "ndcg_linear@10": 0.6725474536030825,
+                "ndcg_exponential@1": 0.561,
+                "ndcg_exponential@5": 0.6571987612384224,
+                "ndcg_exponential@10": 0.6725474536030825,
+                "precision@1": 0.561,
+                "precision@5": 0.1474,
+                "precision@10": 0.07840000000000001,
+                "recall@1": 0.561,
+                "recall@5": 0.737,
+                "recall@10": 0.784,
+                "f1@1": 0.561,
+                "f1@5": 0.24566666666666667,
+                "f1@10": 0.14254545454545456,
+                "map@1": 0.561,
+                "map@5": 0.6304333333333334,
+                "map@10": 0.6368575396825396,
+                "mrr@1": 0.561,
+                "mrr@5": 0.6304333333333334,
+                "mrr@10": 0.6368575396825396,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Visual7W": {
+                "hit@1": 0.585,
+                "hit@5": 0.899,
+                "hit@10": 0.951,
+                "ndcg_linear@1": 0.585,
+                "ndcg_linear@5": 0.7591108361833063,
+                "ndcg_linear@10": 0.7764038115885976,
+                "ndcg_exponential@1": 0.585,
+                "ndcg_exponential@5": 0.7591108361833063,
+                "ndcg_exponential@10": 0.7764038115885976,
+                "precision@1": 0.585,
+                "precision@5": 0.17980000000000002,
+                "precision@10": 0.09510000000000002,
+                "recall@1": 0.585,
+                "recall@5": 0.899,
+                "recall@10": 0.951,
+                "f1@1": 0.585,
+                "f1@5": 0.29966666666666664,
+                "f1@10": 0.17290909090909087,
+                "map@1": 0.585,
+                "map@5": 0.7119166666666666,
+                "map@10": 0.7193325396825396,
+                "mrr@1": 0.585,
+                "mrr@5": 0.7119166666666666,
+                "mrr@10": 0.7193325396825396,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ScienceQA": {
+                "hit@1": 0.473,
+                "hit@5": 0.775,
+                "hit@10": 0.876,
+                "ndcg_linear@1": 0.473,
+                "ndcg_linear@5": 0.637248574258738,
+                "ndcg_linear@10": 0.6701860011263443,
+                "ndcg_exponential@1": 0.473,
+                "ndcg_exponential@5": 0.637248574258738,
+                "ndcg_exponential@10": 0.6701860011263443,
+                "precision@1": 0.473,
+                "precision@5": 0.155,
+                "precision@10": 0.08760000000000001,
+                "recall@1": 0.473,
+                "recall@5": 0.775,
+                "recall@10": 0.876,
+                "f1@1": 0.473,
+                "f1@5": 0.25833333333333336,
+                "f1@10": 0.15927272727272726,
+                "map@1": 0.473,
+                "map@5": 0.5911666666666666,
+                "map@10": 0.604918253968254,
+                "mrr@1": 0.473,
+                "mrr@5": 0.5911666666666666,
+                "mrr@10": 0.604918253968254,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VizWiz": {
+                "hit@1": 0.553,
+                "hit@5": 0.72,
+                "hit@10": 0.749,
+                "ndcg_linear@1": 0.553,
+                "ndcg_linear@5": 0.6469002810865487,
+                "ndcg_linear@10": 0.6563836673734713,
+                "ndcg_exponential@1": 0.553,
+                "ndcg_exponential@5": 0.6469002810865487,
+                "ndcg_exponential@10": 0.6563836673734713,
+                "precision@1": 0.553,
+                "precision@5": 0.144,
+                "precision@10": 0.0749,
+                "recall@1": 0.553,
+                "recall@5": 0.72,
+                "recall@10": 0.749,
+                "f1@1": 0.553,
+                "f1@5": 0.24000000000000005,
+                "f1@10": 0.13618181818181818,
+                "map@1": 0.553,
+                "map@5": 0.6221166666666667,
+                "map@10": 0.6260968253968253,
+                "mrr@1": 0.553,
+                "mrr@5": 0.6221166666666667,
+                "mrr@10": 0.6260968253968253,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "GQA": {
+                "hit@1": 0.744,
+                "hit@5": 0.923,
+                "hit@10": 0.961,
+                "ndcg_linear@1": 0.744,
+                "ndcg_linear@5": 0.8429932967431276,
+                "ndcg_linear@10": 0.8554002697803087,
+                "ndcg_exponential@1": 0.744,
+                "ndcg_exponential@5": 0.8429932967431276,
+                "ndcg_exponential@10": 0.8554002697803087,
+                "precision@1": 0.744,
+                "precision@5": 0.18460000000000001,
+                "precision@10": 0.09610000000000002,
+                "recall@1": 0.744,
+                "recall@5": 0.923,
+                "recall@10": 0.961,
+                "f1@1": 0.744,
+                "f1@5": 0.3076666666666667,
+                "f1@10": 0.17472727272727273,
+                "map@1": 0.744,
+                "map@5": 0.816,
+                "map@10": 0.8211888888888889,
+                "mrr@1": 0.744,
+                "mrr@5": 0.816,
+                "mrr@10": 0.8211888888888889,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "TextVQA": {
+                "hit@1": 0.831,
+                "hit@5": 0.903,
+                "hit@10": 0.92,
+                "ndcg_linear@1": 0.831,
+                "ndcg_linear@5": 0.8717715154114755,
+                "ndcg_linear@10": 0.8772359632419988,
+                "ndcg_exponential@1": 0.831,
+                "ndcg_exponential@5": 0.8717715154114755,
+                "ndcg_exponential@10": 0.8772359632419988,
+                "precision@1": 0.831,
+                "precision@5": 0.1806,
+                "precision@10": 0.09200000000000001,
+                "recall@1": 0.831,
+                "recall@5": 0.903,
+                "recall@10": 0.92,
+                "f1@1": 0.831,
+                "f1@5": 0.301,
+                "f1@10": 0.16727272727272724,
+                "map@1": 0.831,
+                "map@5": 0.8611666666666667,
+                "map@10": 0.8634039682539683,
+                "mrr@1": 0.831,
+                "mrr@5": 0.8611666666666667,
+                "mrr@10": 0.8634039682539683,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VisDial": {
+                "hit@1": 0.794,
+                "hit@5": 0.953,
+                "hit@10": 0.982,
+                "ndcg_linear@1": 0.794,
+                "ndcg_linear@5": 0.8808729209948967,
+                "ndcg_linear@10": 0.8902552237878829,
+                "ndcg_exponential@1": 0.794,
+                "ndcg_exponential@5": 0.8808729209948967,
+                "ndcg_exponential@10": 0.8902552237878829,
+                "precision@1": 0.794,
+                "precision@5": 0.19060000000000002,
+                "precision@10": 0.09820000000000002,
+                "recall@1": 0.794,
+                "recall@5": 0.953,
+                "recall@10": 0.982,
+                "f1@1": 0.794,
+                "f1@5": 0.31766666666666665,
+                "f1@10": 0.1785454545454545,
+                "map@1": 0.794,
+                "map@5": 0.8566333333333334,
+                "map@10": 0.8605126984126985,
+                "mrr@1": 0.794,
+                "mrr@5": 0.8566333333333334,
+                "mrr@10": 0.8605126984126985,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "CIRR": {
+                "hit@1": 0.473,
+                "hit@5": 0.881,
+                "hit@10": 0.933,
+                "ndcg_linear@1": 0.473,
+                "ndcg_linear@5": 0.7007716056263877,
+                "ndcg_linear@10": 0.7176746558999346,
+                "ndcg_exponential@1": 0.473,
+                "ndcg_exponential@5": 0.7007716056263877,
+                "ndcg_exponential@10": 0.7176746558999346,
+                "precision@1": 0.473,
+                "precision@5": 0.1762,
+                "precision@10": 0.09330000000000001,
+                "recall@1": 0.473,
+                "recall@5": 0.881,
+                "recall@10": 0.933,
+                "f1@1": 0.473,
+                "f1@5": 0.2936666666666667,
+                "f1@10": 0.1696363636363636,
+                "map@1": 0.473,
+                "map@5": 0.6398666666666667,
+                "map@10": 0.6468948412698412,
+                "mrr@1": 0.473,
+                "mrr@5": 0.6398666666666667,
+                "mrr@10": 0.6468948412698412,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VisualNews_t2i": {
+                "hit@1": 0.762,
+                "hit@5": 0.884,
+                "hit@10": 0.918,
+                "ndcg_linear@1": 0.762,
+                "ndcg_linear@5": 0.8291412055303354,
+                "ndcg_linear@10": 0.840484153972375,
+                "ndcg_exponential@1": 0.762,
+                "ndcg_exponential@5": 0.8291412055303354,
+                "ndcg_exponential@10": 0.840484153972375,
+                "precision@1": 0.762,
+                "precision@5": 0.17680000000000004,
+                "precision@10": 0.0918,
+                "recall@1": 0.762,
+                "recall@5": 0.884,
+                "recall@10": 0.918,
+                "f1@1": 0.762,
+                "f1@5": 0.29466666666666674,
+                "f1@10": 0.1669090909090909,
+                "map@1": 0.762,
+                "map@5": 0.8107000000000001,
+                "map@10": 0.8155873015873015,
+                "mrr@1": 0.762,
+                "mrr@5": 0.8107000000000001,
+                "mrr@10": 0.8155873015873015,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VisualNews_i2t": {
+                "hit@1": 0.791,
+                "hit@5": 0.908,
+                "hit@10": 0.931,
+                "ndcg_linear@1": 0.791,
+                "ndcg_linear@5": 0.8551690273971704,
+                "ndcg_linear@10": 0.8625508148532727,
+                "ndcg_exponential@1": 0.791,
+                "ndcg_exponential@5": 0.8551690273971704,
+                "ndcg_exponential@10": 0.8625508148532727,
+                "precision@1": 0.791,
+                "precision@5": 0.1816,
+                "precision@10": 0.09310000000000002,
+                "recall@1": 0.791,
+                "recall@5": 0.908,
+                "recall@10": 0.931,
+                "f1@1": 0.791,
+                "f1@5": 0.3026666666666667,
+                "f1@10": 0.16927272727272724,
+                "map@1": 0.791,
+                "map@5": 0.8374166666666667,
+                "map@10": 0.8404353174603174,
+                "mrr@1": 0.791,
+                "mrr@5": 0.8374166666666667,
+                "mrr@10": 0.8404353174603174,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSCOCO_t2i": {
+                "hit@1": 0.741,
+                "hit@5": 0.935,
+                "hit@10": 0.974,
+                "ndcg_linear@1": 0.741,
+                "ndcg_linear@5": 0.8481271424203821,
+                "ndcg_linear@10": 0.8606938083382606,
+                "ndcg_exponential@1": 0.741,
+                "ndcg_exponential@5": 0.8481271424203821,
+                "ndcg_exponential@10": 0.8606938083382606,
+                "precision@1": 0.741,
+                "precision@5": 0.18700000000000003,
+                "precision@10": 0.09740000000000001,
+                "recall@1": 0.741,
+                "recall@5": 0.935,
+                "recall@10": 0.974,
+                "f1@1": 0.741,
+                "f1@5": 0.3116666666666667,
+                "f1@10": 0.17709090909090908,
+                "map@1": 0.741,
+                "map@5": 0.8188666666666666,
+                "map@10": 0.8240309523809524,
+                "mrr@1": 0.741,
+                "mrr@5": 0.8188666666666666,
+                "mrr@10": 0.8240309523809524,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSCOCO_i2t": {
+                "hit@1": 0.697,
+                "hit@5": 0.919,
+                "hit@10": 0.965,
+                "ndcg_linear@1": 0.697,
+                "ndcg_linear@5": 0.8184709214046806,
+                "ndcg_linear@10": 0.833832715681852,
+                "ndcg_exponential@1": 0.697,
+                "ndcg_exponential@5": 0.8184709214046806,
+                "ndcg_exponential@10": 0.833832715681852,
+                "precision@1": 0.697,
+                "precision@5": 0.18380000000000002,
+                "precision@10": 0.09650000000000002,
+                "recall@1": 0.697,
+                "recall@5": 0.919,
+                "recall@10": 0.965,
+                "f1@1": 0.697,
+                "f1@5": 0.30633333333333335,
+                "f1@10": 0.17545454545454545,
+                "map@1": 0.697,
+                "map@5": 0.7847833333333333,
+                "map@10": 0.7914162698412698,
+                "mrr@1": 0.697,
+                "mrr@5": 0.7847833333333333,
+                "mrr@10": 0.7914162698412698,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "NIGHTS": {
+                "hit@1": 0.661,
+                "hit@5": 0.981,
+                "hit@10": 0.995,
+                "ndcg_linear@1": 0.661,
+                "ndcg_linear@5": 0.844481977284671,
+                "ndcg_linear@10": 0.849108971640867,
+                "ndcg_exponential@1": 0.661,
+                "ndcg_exponential@5": 0.844481977284671,
+                "ndcg_exponential@10": 0.849108971640867,
+                "precision@1": 0.661,
+                "precision@5": 0.19620000000000004,
+                "precision@10": 0.09950000000000003,
+                "recall@1": 0.661,
+                "recall@5": 0.981,
+                "recall@10": 0.995,
+                "f1@1": 0.661,
+                "f1@5": 0.327,
+                "f1@10": 0.18090909090909088,
+                "map@1": 0.661,
+                "map@5": 0.7978666666666667,
+                "map@10": 0.7998357142857143,
+                "mrr@1": 0.661,
+                "mrr@5": 0.7978666666666667,
+                "mrr@10": 0.7998357142857143,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "WebQA": {
+                "hit@1": 0.911,
+                "hit@5": 0.989,
+                "hit@10": 0.993,
+                "ndcg_linear@1": 0.911,
+                "ndcg_linear@5": 0.9562555664516004,
+                "ndcg_linear@10": 0.9574560942638872,
+                "ndcg_exponential@1": 0.911,
+                "ndcg_exponential@5": 0.9562555664516004,
+                "ndcg_exponential@10": 0.9574560942638872,
+                "precision@1": 0.911,
+                "precision@5": 0.1978,
+                "precision@10": 0.09930000000000001,
+                "recall@1": 0.911,
+                "recall@5": 0.989,
+                "recall@10": 0.993,
+                "f1@1": 0.911,
+                "f1@5": 0.32966666666666666,
+                "f1@10": 0.18054545454545456,
+                "map@1": 0.911,
+                "map@5": 0.94505,
+                "map@10": 0.9454928571428572,
+                "mrr@1": 0.911,
+                "mrr@5": 0.94505,
+                "mrr@10": 0.9454928571428572,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "FashionIQ": {
+                "hit@1": 0.185,
+                "hit@5": 0.438,
+                "hit@10": 0.546,
+                "ndcg_linear@1": 0.185,
+                "ndcg_linear@5": 0.31634442487309344,
+                "ndcg_linear@10": 0.3509652493511108,
+                "ndcg_exponential@1": 0.185,
+                "ndcg_exponential@5": 0.31634442487309344,
+                "ndcg_exponential@10": 0.3509652493511108,
+                "precision@1": 0.185,
+                "precision@5": 0.0876,
+                "precision@10": 0.054599999999999996,
+                "recall@1": 0.185,
+                "recall@5": 0.438,
+                "recall@10": 0.546,
+                "f1@1": 0.185,
+                "f1@5": 0.14600000000000005,
+                "f1@10": 0.09927272727272728,
+                "map@1": 0.185,
+                "map@5": 0.27613333333333334,
+                "map@10": 0.2902329365079365,
+                "mrr@1": 0.185,
+                "mrr@5": 0.27613333333333334,
+                "mrr@10": 0.2902329365079365,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Wiki-SS-NQ": {
+                "hit@1": 0.688,
+                "hit@5": 0.879,
+                "hit@10": 0.917,
+                "ndcg_linear@1": 0.688,
+                "ndcg_linear@5": 0.7938577226380621,
+                "ndcg_linear@10": 0.8061437893498671,
+                "ndcg_exponential@1": 0.688,
+                "ndcg_exponential@5": 0.7938577226380621,
+                "ndcg_exponential@10": 0.8061437893498671,
+                "precision@1": 0.688,
+                "precision@5": 0.1758,
+                "precision@10": 0.09170000000000002,
+                "recall@1": 0.688,
+                "recall@5": 0.879,
+                "recall@10": 0.917,
+                "f1@1": 0.688,
+                "f1@5": 0.293,
+                "f1@10": 0.16672727272727272,
+                "map@1": 0.688,
+                "map@5": 0.7651333333333334,
+                "map@10": 0.7702043650793651,
+                "mrr@1": 0.688,
+                "mrr@5": 0.7651333333333334,
+                "mrr@10": 0.7702043650793651,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "OVEN": {
+                "hit@1": 0.691,
+                "hit@5": 0.873,
+                "hit@10": 0.921,
+                "ndcg_linear@1": 0.691,
+                "ndcg_linear@5": 0.7901376964257225,
+                "ndcg_linear@10": 0.8062841721039863,
+                "ndcg_exponential@1": 0.691,
+                "ndcg_exponential@5": 0.7901376964257225,
+                "ndcg_exponential@10": 0.8062841721039863,
+                "precision@1": 0.691,
+                "precision@5": 0.17460000000000003,
+                "precision@10": 0.09210000000000002,
+                "recall@1": 0.691,
+                "recall@5": 0.873,
+                "recall@10": 0.921,
+                "f1@1": 0.691,
+                "f1@5": 0.291,
+                "f1@10": 0.16745454545454544,
+                "map@1": 0.691,
+                "map@5": 0.7623333333333333,
+                "map@10": 0.7693662698412698,
+                "mrr@1": 0.691,
+                "mrr@5": 0.7623333333333333,
+                "mrr@10": 0.7693662698412698,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "EDIS": {
+                "hit@1": 0.906,
+                "hit@5": 0.981,
+                "hit@10": 0.991,
+                "ndcg_linear@1": 0.906,
+                "ndcg_linear@5": 0.9501104422904603,
+                "ndcg_linear@10": 0.9532607983085303,
+                "ndcg_exponential@1": 0.906,
+                "ndcg_exponential@5": 0.9501104422904603,
+                "ndcg_exponential@10": 0.9532607983085303,
+                "precision@1": 0.906,
+                "precision@5": 0.19620000000000004,
+                "precision@10": 0.09910000000000002,
+                "recall@1": 0.906,
+                "recall@5": 0.981,
+                "recall@10": 0.991,
+                "f1@1": 0.906,
+                "f1@5": 0.327,
+                "f1@10": 0.18018181818181817,
+                "map@1": 0.906,
+                "map@5": 0.93945,
+                "map@10": 0.9407003968253969,
+                "mrr@1": 0.906,
+                "mrr@5": 0.93945,
+                "mrr@10": 0.9407003968253969,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSCOCO": {
+                "hit@1": 0.668,
+                "hit@5": 0.83,
+                "hit@10": 0.865,
+                "ndcg_linear@1": 0.668,
+                "ndcg_linear@5": 0.7581674826240974,
+                "ndcg_linear@10": 0.7697284046193552,
+                "ndcg_exponential@1": 0.668,
+                "ndcg_exponential@5": 0.7581674826240974,
+                "ndcg_exponential@10": 0.7697284046193552,
+                "precision@1": 0.668,
+                "precision@5": 0.166,
+                "precision@10": 0.0865,
+                "recall@1": 0.668,
+                "recall@5": 0.83,
+                "recall@10": 0.865,
+                "f1@1": 0.668,
+                "f1@5": 0.2766666666666667,
+                "f1@10": 0.15727272727272726,
+                "map@1": 0.668,
+                "map@5": 0.7338833333333334,
+                "map@10": 0.7387964285714285,
+                "mrr@1": 0.668,
+                "mrr@5": 0.7338833333333334,
+                "mrr@10": 0.7387964285714285,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "RefCOCO": {
+                "hit@1": 0.841,
+                "hit@5": 0.972,
+                "hit@10": 0.985,
+                "ndcg_linear@1": 0.841,
+                "ndcg_linear@5": 0.9172700496870271,
+                "ndcg_linear@10": 0.9215638033311572,
+                "ndcg_exponential@1": 0.841,
+                "ndcg_exponential@5": 0.9172700496870271,
+                "ndcg_exponential@10": 0.9215638033311572,
+                "precision@1": 0.841,
+                "precision@5": 0.19440000000000004,
+                "precision@10": 0.09850000000000003,
+                "recall@1": 0.841,
+                "recall@5": 0.972,
+                "recall@10": 0.985,
+                "f1@1": 0.841,
+                "f1@5": 0.324,
+                "f1@10": 0.17909090909090908,
+                "map@1": 0.841,
+                "map@5": 0.8985333333333333,
+                "map@10": 0.9003567460317461,
+                "mrr@1": 0.841,
+                "mrr@5": 0.8985333333333333,
+                "mrr@10": 0.9003567460317461,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "RefCOCO-Matching": {
+                "hit@1": 0.879,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.879,
+                "ndcg_linear@5": 0.9572352894428607,
+                "ndcg_linear@10": 0.9572352894428607,
+                "ndcg_exponential@1": 0.879,
+                "ndcg_exponential@5": 0.9572352894428607,
+                "ndcg_exponential@10": 0.9572352894428607,
+                "precision@1": 0.879,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.879,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.879,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.879,
+                "map@5": 0.9425,
+                "map@10": 0.9425,
+                "mrr@1": 0.879,
+                "mrr@5": 0.9395,
+                "mrr@10": 0.9395,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Visual7W-Pointing": {
+                "hit@1": 0.846,
+                "hit@5": 0.949,
+                "hit@10": 0.976,
+                "ndcg_linear@1": 0.846,
+                "ndcg_linear@5": 0.905021040236476,
+                "ndcg_linear@10": 0.9137472283052835,
+                "ndcg_exponential@1": 0.846,
+                "ndcg_exponential@5": 0.905021040236476,
+                "ndcg_exponential@10": 0.9137472283052835,
+                "precision@1": 0.846,
+                "precision@5": 0.18980000000000002,
+                "precision@10": 0.0976,
+                "recall@1": 0.846,
+                "recall@5": 0.949,
+                "recall@10": 0.976,
+                "f1@1": 0.846,
+                "f1@5": 0.3163333333333333,
+                "f1@10": 0.17745454545454545,
+                "map@1": 0.846,
+                "map@5": 0.89,
+                "map@10": 0.8936003968253967,
+                "mrr@1": 0.846,
+                "mrr@5": 0.89,
+                "mrr@10": 0.8936003968253967,
+                "num_pred": 1000,
+                "num_data": 1000
+            }
+        },
+        "visdoc": {
+            "ViDoRe_arxivqa": {
+                "hit@1": 0.674,
+                "hit@5": 0.796,
+                "hit@10": 0.842,
+                "ndcg_linear@1": 0.674,
+                "ndcg_linear@5": 0.7371517581853998,
+                "ndcg_linear@10": 0.7522168662953272,
+                "ndcg_exponential@1": 0.674,
+                "ndcg_exponential@5": 0.7371517581853998,
+                "ndcg_exponential@10": 0.7522168662953272,
+                "precision@1": 0.674,
+                "precision@5": 0.15919999999999998,
+                "precision@10": 0.0842,
+                "recall@1": 0.674,
+                "recall@5": 0.796,
+                "recall@10": 0.842,
+                "f1@1": 0.674,
+                "f1@5": 0.26533333333333337,
+                "f1@10": 0.15309090909090908,
+                "map@1": 0.674,
+                "map@5": 0.7178000000000001,
+                "map@10": 0.7241246031746031,
+                "mrr@1": 0.674,
+                "mrr@5": 0.7178000000000001,
+                "mrr@10": 0.7241246031746031,
+                "num_pred": 500,
+                "num_data": 500
+            },
+            "ViDoRe_docvqa": {
+                "hit@1": 0.35698447893569846,
+                "hit@5": 0.5343680709534369,
+                "hit@10": 0.6053215077605322,
+                "ndcg_linear@1": 0.35698447893569846,
+                "ndcg_linear@5": 0.450097536929264,
+                "ndcg_linear@10": 0.4728021762728742,
+                "ndcg_exponential@1": 0.35698447893569846,
+                "ndcg_exponential@5": 0.450097536929264,
+                "ndcg_exponential@10": 0.4728021762728742,
+                "precision@1": 0.35698447893569846,
+                "precision@5": 0.1082039911308204,
+                "precision@10": 0.06186252771618625,
+                "recall@1": 0.3526237989652624,
+                "recall@5": 0.5276053215077605,
+                "recall@10": 0.5964153732446416,
+                "f1@1": 0.35365853658536583,
+                "f1@5": 0.17856304384281432,
+                "f1@10": 0.11150512458716451,
+                "map@1": 0.35698447893569846,
+                "map@5": 0.42237620103473766,
+                "map@10": 0.4318832928589026,
+                "mrr@1": 0.35698447893569846,
+                "mrr@5": 0.42738359201773835,
+                "mrr@10": 0.4369241192411924,
+                "num_pred": 451,
+                "num_data": 451
+            },
+            "ViDoRe_infovqa": {
+                "hit@1": 0.7348178137651822,
+                "hit@5": 0.8805668016194332,
+                "hit@10": 0.9230769230769231,
+                "ndcg_linear@1": 0.7348178137651822,
+                "ndcg_linear@5": 0.8141460784346112,
+                "ndcg_linear@10": 0.8280697125910069,
+                "ndcg_exponential@1": 0.7348178137651822,
+                "ndcg_exponential@5": 0.8141460784346112,
+                "ndcg_exponential@10": 0.8280697125910069,
+                "precision@1": 0.7348178137651822,
+                "precision@5": 0.17651821862348177,
+                "precision@10": 0.09251012145748988,
+                "recall@1": 0.7348178137651822,
+                "recall@5": 0.8805668016194332,
+                "recall@10": 0.9230769230769231,
+                "f1@1": 0.7348178137651822,
+                "f1@5": 0.29400424137266246,
+                "f1@10": 0.16813887866519445,
+                "map@1": 0.7348178137651822,
+                "map@5": 0.7919028340080971,
+                "map@10": 0.7977612299980721,
+                "mrr@1": 0.7348178137651822,
+                "mrr@5": 0.7917341430499325,
+                "mrr@10": 0.7975925390399075,
+                "num_pred": 494,
+                "num_data": 494
+            },
+            "ViDoRe_tabfquad": {
+                "hit@1": 0.8035714285714286,
+                "hit@5": 0.95,
+                "hit@10": 0.9785714285714285,
+                "ndcg_linear@1": 0.8035714285714286,
+                "ndcg_linear@5": 0.8880525993445776,
+                "ndcg_linear@10": 0.8974326459361368,
+                "ndcg_exponential@1": 0.8035714285714286,
+                "ndcg_exponential@5": 0.8880525993445776,
+                "ndcg_exponential@10": 0.8974326459361368,
+                "precision@1": 0.8035714285714286,
+                "precision@5": 0.18999999999999995,
+                "precision@10": 0.09785714285714285,
+                "recall@1": 0.8035714285714286,
+                "recall@5": 0.95,
+                "recall@10": 0.9785714285714285,
+                "f1@1": 0.8035714285714286,
+                "f1@5": 0.3166666666666667,
+                "f1@10": 0.17792207792207795,
+                "map@1": 0.8035714285714286,
+                "map@5": 0.866904761904762,
+                "map@10": 0.8708616780045352,
+                "mrr@1": 0.8035714285714286,
+                "mrr@5": 0.866904761904762,
+                "mrr@10": 0.8708616780045352,
+                "num_pred": 280,
+                "num_data": 280
+            },
+            "ViDoRe_tatdqa": {
+                "hit@1": 0.35297691373025514,
+                "hit@5": 0.6233292831105711,
+                "hit@10": 0.7424058323207776,
+                "ndcg_linear@1": 0.35297691373025514,
+                "ndcg_linear@5": 0.4959176191553392,
+                "ndcg_linear@10": 0.5340857919511109,
+                "ndcg_exponential@1": 0.35297691373025514,
+                "ndcg_exponential@5": 0.4959176191553392,
+                "ndcg_exponential@10": 0.5340857919511109,
+                "precision@1": 0.35297691373025514,
+                "precision@5": 0.12503037667071687,
+                "precision@10": 0.07448359659781288,
+                "recall@1": 0.35236938031591736,
+                "recall@5": 0.6230255164034022,
+                "recall@10": 0.7424058323207776,
+                "f1@1": 0.35257189145402995,
+                "f1@5": 0.20818144997974894,
+                "f1@10": 0.13535108067307342,
+                "map@1": 0.35297691373025514,
+                "map@5": 0.45365026326447955,
+                "map@10": 0.4691597234276456,
+                "mrr@1": 0.35297691373025514,
+                "mrr@5": 0.4537262049412718,
+                "mrr@10": 0.4691597234276456,
+                "num_pred": 1646,
+                "num_data": 1646
+            },
+            "ViDoRe_shiftproject": {
+                "hit@1": 0.57,
+                "hit@5": 0.87,
+                "hit@10": 0.91,
+                "ndcg_linear@1": 0.57,
+                "ndcg_linear@5": 0.7257169786666456,
+                "ndcg_linear@10": 0.7388716274345029,
+                "ndcg_exponential@1": 0.57,
+                "ndcg_exponential@5": 0.7257169786666456,
+                "ndcg_exponential@10": 0.7388716274345029,
+                "precision@1": 0.57,
+                "precision@5": 0.17399999999999996,
+                "precision@10": 0.09099999999999998,
+                "recall@1": 0.57,
+                "recall@5": 0.87,
+                "recall@10": 0.91,
+                "f1@1": 0.57,
+                "f1@5": 0.29000000000000004,
+                "f1@10": 0.16545454545454555,
+                "map@1": 0.57,
+                "map@5": 0.6779999999999999,
+                "map@10": 0.6835357142857142,
+                "mrr@1": 0.57,
+                "mrr@5": 0.6779999999999999,
+                "mrr@10": 0.6835357142857142,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_artificial_intelligence": {
+                "hit@1": 0.84,
+                "hit@5": 0.95,
+                "hit@10": 0.98,
+                "ndcg_linear@1": 0.84,
+                "ndcg_linear@5": 0.8979742462875491,
+                "ndcg_linear@10": 0.90802430025982,
+                "ndcg_exponential@1": 0.84,
+                "ndcg_exponential@5": 0.8979742462875491,
+                "ndcg_exponential@10": 0.90802430025982,
+                "precision@1": 0.84,
+                "precision@5": 0.18999999999999997,
+                "precision@10": 0.09799999999999999,
+                "recall@1": 0.84,
+                "recall@5": 0.95,
+                "recall@10": 0.98,
+                "f1@1": 0.84,
+                "f1@5": 0.3166666666666667,
+                "f1@10": 0.17818181818181825,
+                "map@1": 0.84,
+                "map@5": 0.8806666666666665,
+                "map@10": 0.8850119047619047,
+                "mrr@1": 0.84,
+                "mrr@5": 0.8806666666666665,
+                "mrr@10": 0.8850119047619047,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_energy": {
+                "hit@1": 0.75,
+                "hit@5": 0.92,
+                "hit@10": 0.94,
+                "ndcg_linear@1": 0.75,
+                "ndcg_linear@5": 0.8426960276192014,
+                "ndcg_linear@10": 0.8489200092157135,
+                "ndcg_exponential@1": 0.75,
+                "ndcg_exponential@5": 0.8426960276192014,
+                "ndcg_exponential@10": 0.8489200092157135,
+                "precision@1": 0.75,
+                "precision@5": 0.18399999999999994,
+                "precision@10": 0.09399999999999999,
+                "recall@1": 0.75,
+                "recall@5": 0.92,
+                "recall@10": 0.94,
+                "f1@1": 0.75,
+                "f1@5": 0.3066666666666667,
+                "f1@10": 0.170909090909091,
+                "map@1": 0.75,
+                "map@5": 0.8168333333333332,
+                "map@10": 0.8192619047619047,
+                "mrr@1": 0.75,
+                "mrr@5": 0.8168333333333332,
+                "mrr@10": 0.8192619047619047,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_government_reports": {
+                "hit@1": 0.75,
+                "hit@5": 0.96,
+                "hit@10": 0.98,
+                "ndcg_linear@1": 0.75,
+                "ndcg_linear@5": 0.8706343991257575,
+                "ndcg_linear@10": 0.8769780324157306,
+                "ndcg_exponential@1": 0.75,
+                "ndcg_exponential@5": 0.8706343991257575,
+                "ndcg_exponential@10": 0.8769780324157306,
+                "precision@1": 0.75,
+                "precision@5": 0.19199999999999995,
+                "precision@10": 0.09799999999999999,
+                "recall@1": 0.75,
+                "recall@5": 0.96,
+                "recall@10": 0.98,
+                "f1@1": 0.75,
+                "f1@5": 0.32000000000000006,
+                "f1@10": 0.17818181818181825,
+                "map@1": 0.75,
+                "map@5": 0.84,
+                "map@10": 0.8425396825396825,
+                "mrr@1": 0.75,
+                "mrr@5": 0.84,
+                "mrr@10": 0.8425396825396825,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_healthcare_industry": {
+                "hit@1": 0.82,
+                "hit@5": 0.99,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.82,
+                "ndcg_linear@5": 0.9164460946182851,
+                "ndcg_linear@10": 0.9200081664893653,
+                "ndcg_exponential@1": 0.82,
+                "ndcg_exponential@5": 0.9164460946182851,
+                "ndcg_exponential@10": 0.9200081664893653,
+                "precision@1": 0.82,
+                "precision@5": 0.19799999999999998,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.82,
+                "recall@5": 0.99,
+                "recall@10": 1.0,
+                "f1@1": 0.82,
+                "f1@5": 0.3300000000000001,
+                "f1@10": 0.1818181818181819,
+                "map@1": 0.82,
+                "map@5": 0.8915000000000001,
+                "map@10": 0.8931666666666667,
+                "mrr@1": 0.82,
+                "mrr@5": 0.8915000000000001,
+                "mrr@10": 0.8931666666666667,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_esg_reports_human_labeled_v2": {
+                "hit@1": 0.5384615384615384,
+                "hit@5": 0.8076923076923077,
+                "hit@10": 0.9038461538461539,
+                "ndcg_linear@1": 0.5384615384615384,
+                "ndcg_linear@5": 0.5857101818076832,
+                "ndcg_linear@10": 0.6265239767252975,
+                "ndcg_exponential@1": 0.5384615384615384,
+                "ndcg_exponential@5": 0.5878008188023434,
+                "ndcg_exponential@10": 0.6292190672539935,
+                "precision@1": 0.5384615384615384,
+                "precision@5": 0.2269230769230769,
+                "precision@10": 0.14423076923076922,
+                "recall@1": 0.3721153846153846,
+                "recall@5": 0.6419955044955046,
+                "recall@10": 0.7564185814185814,
+                "f1@1": 0.4093688362919133,
+                "f1@5": 0.3009682719241543,
+                "f1@10": 0.22103693817494724,
+                "map@1": 0.5384615384615384,
+                "map@5": 0.5227403846153846,
+                "map@10": 0.5419889601139601,
+                "mrr@1": 0.5384615384615384,
+                "mrr@5": 0.6317307692307691,
+                "mrr@10": 0.6435363247863248,
+                "num_pred": 52,
+                "num_data": 52
+            },
+            "ViDoRe_biomedical_lectures_v2": {
+                "hit@1": 0.51875,
+                "hit@5": 0.81875,
+                "hit@10": 0.90625,
+                "ndcg_linear@1": 0.51875,
+                "ndcg_linear@5": 0.5599127423800386,
+                "ndcg_linear@10": 0.6024210971755942,
+                "ndcg_exponential@1": 0.51875,
+                "ndcg_exponential@5": 0.5599127423800386,
+                "ndcg_exponential@10": 0.6024210971755943,
+                "precision@1": 0.51875,
+                "precision@5": 0.255,
+                "precision@10": 0.169375,
+                "recall@1": 0.31672416125541125,
+                "recall@5": 0.5947338537687802,
+                "recall@10": 0.7137503580977846,
+                "f1@1": 0.3591993596681097,
+                "f1@5": 0.3177249382561883,
+                "f1@10": 0.24713793423352243,
+                "map@1": 0.51875,
+                "map@5": 0.4872517361111111,
+                "map@10": 0.5087469706632654,
+                "mrr@1": 0.51875,
+                "mrr@5": 0.6339583333333334,
+                "mrr@10": 0.6465128968253968,
+                "num_pred": 160,
+                "num_data": 160
+            },
+            "ViDoRe_biomedical_lectures_v2_multilingual": {
+                "hit@1": 0.4875,
+                "hit@5": 0.796875,
+                "hit@10": 0.8734375,
+                "ndcg_linear@1": 0.4875,
+                "ndcg_linear@5": 0.5287109563499492,
+                "ndcg_linear@10": 0.5674601302198331,
+                "ndcg_exponential@1": 0.4875,
+                "ndcg_exponential@5": 0.5287109563499492,
+                "ndcg_exponential@10": 0.5674601302198332,
+                "precision@1": 0.4875,
+                "precision@5": 0.2425,
+                "precision@10": 0.16031250000000002,
+                "recall@1": 0.3014277259199134,
+                "recall@5": 0.567796524859944,
+                "recall@10": 0.679209887079832,
+                "f1@1": 0.3399399125180375,
+                "f1@5": 0.30168018093018095,
+                "f1@10": 0.23434101101104776,
+                "map@1": 0.4875,
+                "map@5": 0.45663975694444436,
+                "map@10": 0.476485510263133,
+                "mrr@1": 0.4875,
+                "mrr@5": 0.5992447916666667,
+                "mrr@10": 0.6098208085317461,
+                "num_pred": 640,
+                "num_data": 640
+            },
+            "ViDoRe_economics_reports_v2": {
+                "hit@1": 0.603448275862069,
+                "hit@5": 0.9137931034482759,
+                "hit@10": 0.9655172413793104,
+                "ndcg_linear@1": 0.603448275862069,
+                "ndcg_linear@5": 0.5638400234406225,
+                "ndcg_linear@10": 0.5396968611894655,
+                "ndcg_exponential@1": 0.603448275862069,
+                "ndcg_exponential@5": 0.5638400234406225,
+                "ndcg_exponential@10": 0.5396968611894654,
+                "precision@1": 0.603448275862069,
+                "precision@5": 0.48620689655172417,
+                "precision@10": 0.38620689655172413,
+                "recall@1": 0.07922120644047585,
+                "recall@5": 0.31144897230589963,
+                "recall@10": 0.420215949960037,
+                "f1@1": 0.12320812368054741,
+                "f1@5": 0.3051673741988237,
+                "f1@10": 0.3255100111938643,
+                "map@1": 0.603448275862069,
+                "map@5": 0.45996647509578553,
+                "map@10": 0.4012958157400891,
+                "mrr@1": 0.603448275862069,
+                "mrr@5": 0.7333333333333334,
+                "mrr@10": 0.7408045977011496,
+                "num_pred": 58,
+                "num_data": 58
+            },
+            "ViDoRe_economics_reports_v2_multilingual": {
+                "hit@1": 0.4956896551724138,
+                "hit@5": 0.8405172413793104,
+                "hit@10": 0.9051724137931034,
+                "ndcg_linear@1": 0.4956896551724138,
+                "ndcg_linear@5": 0.47894537575268276,
+                "ndcg_linear@10": 0.4634894340243453,
+                "ndcg_exponential@1": 0.4956896551724138,
+                "ndcg_exponential@5": 0.47894537575268276,
+                "ndcg_exponential@10": 0.46348943402434517,
+                "precision@1": 0.4956896551724138,
+                "precision@5": 0.4241379310344828,
+                "precision@10": 0.34310344827586203,
+                "recall@1": 0.061453691971240614,
+                "recall@5": 0.2611222491517723,
+                "recall@10": 0.3668674404931723,
+                "f1@1": 0.09778602775368742,
+                "f1@5": 0.2569545227156983,
+                "f1@10": 0.285174136727715,
+                "map@1": 0.4956896551724138,
+                "map@5": 0.3815541187739463,
+                "map@10": 0.33134604676001944,
+                "mrr@1": 0.4956896551724138,
+                "mrr@5": 0.626580459770115,
+                "mrr@10": 0.6353721948549534,
+                "num_pred": 232,
+                "num_data": 232
+            },
+            "ViDoRe_esg_reports_v2": {
+                "hit@1": 0.5789473684210527,
+                "hit@5": 0.7894736842105263,
+                "hit@10": 0.9122807017543859,
+                "ndcg_linear@1": 0.5789473684210527,
+                "ndcg_linear@5": 0.5578170329058069,
+                "ndcg_linear@10": 0.6128892612884805,
+                "ndcg_exponential@1": 0.5789473684210527,
+                "ndcg_exponential@5": 0.5578170329058068,
+                "ndcg_exponential@10": 0.6128892612884805,
+                "precision@1": 0.5789473684210527,
+                "precision@5": 0.29122807017543856,
+                "precision@10": 0.20175438596491233,
+                "recall@1": 0.25954469507101086,
+                "recall@5": 0.5711779448621553,
+                "recall@10": 0.7492481203007518,
+                "f1@1": 0.3216462491598608,
+                "f1@5": 0.3334364669370541,
+                "f1@10": 0.2802500048548606,
+                "map@1": 0.5789473684210527,
+                "map@5": 0.47171052631578947,
+                "map@10": 0.49674951267056533,
+                "mrr@1": 0.5789473684210527,
+                "mrr@5": 0.6605263157894736,
+                "mrr@10": 0.6788707880813143,
+                "num_pred": 57,
+                "num_data": 57
+            },
+            "ViDoRe_esg_reports_v2_multilingual": {
+                "hit@1": 0.5394736842105263,
+                "hit@5": 0.793859649122807,
+                "hit@10": 0.8903508771929824,
+                "ndcg_linear@1": 0.5394736842105263,
+                "ndcg_linear@5": 0.5339065136145643,
+                "ndcg_linear@10": 0.5850795107813987,
+                "ndcg_exponential@1": 0.5394736842105263,
+                "ndcg_exponential@5": 0.5339065136145642,
+                "ndcg_exponential@10": 0.5850795107813987,
+                "precision@1": 0.5394736842105263,
+                "precision@5": 0.27017543859649124,
+                "precision@10": 0.18815789473684214,
+                "recall@1": 0.2494935254803676,
+                "recall@5": 0.5548819966583124,
+                "recall@10": 0.7172044695071011,
+                "f1@1": 0.3067280776627419,
+                "f1@5": 0.3151451349460318,
+                "f1@10": 0.2614781609210891,
+                "map@1": 0.5394736842105263,
+                "map@5": 0.443368664717349,
+                "map@10": 0.46742985937064885,
+                "mrr@1": 0.5394736842105263,
+                "mrr@5": 0.6444444444444444,
+                "mrr@10": 0.6574578808131439,
+                "num_pred": 228,
+                "num_data": 228
+            },
+            "VisRAG_ArxivQA": {
+                "hit@1": 0.6801470588235294,
+                "hit@5": 0.7879901960784313,
+                "hit@10": 0.8247549019607843,
+                "ndcg_linear@1": 0.6801470588235294,
+                "ndcg_linear@5": 0.738423381102982,
+                "ndcg_linear@10": 0.7502580518543612,
+                "ndcg_exponential@1": 0.6801470588235294,
+                "ndcg_exponential@5": 0.738423381102982,
+                "ndcg_exponential@10": 0.7502580518543612,
+                "precision@1": 0.6801470588235294,
+                "precision@5": 0.15759803921568627,
+                "precision@10": 0.08247549019607842,
+                "recall@1": 0.6801470588235294,
+                "recall@5": 0.7879901960784313,
+                "recall@10": 0.8247549019607843,
+                "f1@1": 0.6801470588235294,
+                "f1@5": 0.2626633986928105,
+                "f1@10": 0.14995543672014264,
+                "map@1": 0.6801470588235294,
+                "map@5": 0.7218545751633987,
+                "map@10": 0.7267113095238096,
+                "mrr@1": 0.6801470588235294,
+                "mrr@5": 0.7218545751633987,
+                "mrr@10": 0.7267113095238096,
+                "num_pred": 816,
+                "num_data": 816
+            },
+            "VisRAG_ChartQA": {
+                "hit@1": 0.7142857142857143,
+                "hit@5": 0.873015873015873,
+                "hit@10": 0.9047619047619048,
+                "ndcg_linear@1": 0.7142857142857143,
+                "ndcg_linear@5": 0.8017188056146399,
+                "ndcg_linear@10": 0.813026970284736,
+                "ndcg_exponential@1": 0.7142857142857143,
+                "ndcg_exponential@5": 0.8017188056146399,
+                "ndcg_exponential@10": 0.813026970284736,
+                "precision@1": 0.7142857142857143,
+                "precision@5": 0.1746031746031745,
+                "precision@10": 0.09047619047619043,
+                "recall@1": 0.7142857142857143,
+                "recall@5": 0.873015873015873,
+                "recall@10": 0.9047619047619048,
+                "f1@1": 0.7142857142857143,
+                "f1@5": 0.29100529100529093,
+                "f1@10": 0.16450216450216454,
+                "map@1": 0.7142857142857143,
+                "map@5": 0.7777777777777778,
+                "map@10": 0.7830687830687831,
+                "mrr@1": 0.7142857142857143,
+                "mrr@5": 0.7777777777777778,
+                "mrr@10": 0.7830687830687831,
+                "num_pred": 63,
+                "num_data": 63
+            },
+            "VisRAG_MP-DocVQA": {
+                "hit@1": 0.6412859560067682,
+                "hit@5": 0.8426395939086294,
+                "hit@10": 0.8917089678510999,
+                "ndcg_linear@1": 0.6412859560067682,
+                "ndcg_linear@5": 0.7505637234725795,
+                "ndcg_linear@10": 0.766968360939859,
+                "ndcg_exponential@1": 0.6412859560067682,
+                "ndcg_exponential@5": 0.7505637234725795,
+                "ndcg_exponential@10": 0.766968360939859,
+                "precision@1": 0.6412859560067682,
+                "precision@5": 0.16852791878172588,
+                "precision@10": 0.08917089678510996,
+                "recall@1": 0.6412859560067682,
+                "recall@5": 0.8426395939086294,
+                "recall@10": 0.8917089678510999,
+                "f1@1": 0.6412859560067682,
+                "f1@5": 0.28087986463620984,
+                "f1@10": 0.16212890324565454,
+                "map@1": 0.6412859560067682,
+                "map@5": 0.7197123519458545,
+                "map@10": 0.7268001504042113,
+                "mrr@1": 0.6412859560067682,
+                "mrr@5": 0.7197123519458545,
+                "mrr@10": 0.7268001504042113,
+                "num_pred": 591,
+                "num_data": 591
+            },
+            "VisRAG_SlideVQA": {
+                "hit@1": 0.8525179856115108,
+                "hit@5": 0.9586330935251799,
+                "hit@10": 0.9730215827338129,
+                "ndcg_linear@1": 0.8525179856115108,
+                "ndcg_linear@5": 0.891546429115487,
+                "ndcg_linear@10": 0.89899357256547,
+                "ndcg_exponential@1": 0.8525179856115108,
+                "ndcg_exponential@5": 0.891546429115487,
+                "ndcg_exponential@10": 0.89899357256547,
+                "precision@1": 0.8525179856115108,
+                "precision@5": 0.23489208633093525,
+                "precision@10": 0.12032374100719423,
+                "recall@1": 0.7320143884892086,
+                "recall@5": 0.9370503597122302,
+                "recall@10": 0.9577338129496403,
+                "f1@1": 0.7721822541966427,
+                "f1@5": 0.3688763275094211,
+                "f1@10": 0.2113309352517986,
+                "map@1": 0.8525179856115108,
+                "map@5": 0.8655125899280575,
+                "map@10": 0.8694287427201095,
+                "mrr@1": 0.8525179856115108,
+                "mrr@5": 0.8969724220623502,
+                "mrr@10": 0.899004367934224,
+                "num_pred": 556,
+                "num_data": 556
+            },
+            "VisRAG_InfoVQA": {
+                "hit@1": 0.7841225626740947,
+                "hit@5": 0.9303621169916435,
+                "hit@10": 0.9623955431754875,
+                "ndcg_linear@1": 0.7841225626740947,
+                "ndcg_linear@5": 0.8666397408730767,
+                "ndcg_linear@10": 0.877368279907231,
+                "ndcg_exponential@1": 0.7841225626740947,
+                "ndcg_exponential@5": 0.8666397408730767,
+                "ndcg_exponential@10": 0.877368279907231,
+                "precision@1": 0.7841225626740947,
+                "precision@5": 0.18607242339832863,
+                "precision@10": 0.09623955431754874,
+                "recall@1": 0.7841225626740947,
+                "recall@5": 0.9303621169916435,
+                "recall@10": 0.9623955431754875,
+                "f1@1": 0.7841225626740947,
+                "f1@5": 0.3101207056638813,
+                "f1@10": 0.1749810078500887,
+                "map@1": 0.7841225626740947,
+                "map@5": 0.8450557103064067,
+                "map@10": 0.8497004465667417,
+                "mrr@1": 0.7841225626740947,
+                "mrr@5": 0.8450557103064067,
+                "mrr@10": 0.8497004465667417,
+                "num_pred": 718,
+                "num_data": 718
+            },
+            "VisRAG_PlotQA": {
+                "hit@1": 0.4646581691772885,
+                "hit@5": 0.7392815758980301,
+                "hit@10": 0.8331402085747392,
+                "ndcg_linear@1": 0.4646581691772885,
+                "ndcg_linear@5": 0.6099250068225089,
+                "ndcg_linear@10": 0.6403291784219023,
+                "ndcg_exponential@1": 0.4646581691772885,
+                "ndcg_exponential@5": 0.6099250068225089,
+                "ndcg_exponential@10": 0.6403291784219023,
+                "precision@1": 0.4646581691772885,
+                "precision@5": 0.14785631517960598,
+                "precision@10": 0.0833140208574739,
+                "recall@1": 0.4646581691772885,
+                "recall@5": 0.7392815758980301,
+                "recall@10": 0.8331402085747392,
+                "f1@1": 0.4646581691772885,
+                "f1@5": 0.2464271919660101,
+                "f1@10": 0.1514800379226799,
+                "map@1": 0.4646581691772885,
+                "map@5": 0.5669949787562766,
+                "map@10": 0.57957475767441,
+                "mrr@1": 0.4646581691772885,
+                "mrr@5": 0.5669949787562766,
+                "mrr@10": 0.57957475767441,
+                "num_pred": 863,
+                "num_data": 863
+            },
+            "ViDoSeek-page": {
+                "hit@1": 0.023642732049036778,
+                "hit@5": 0.4001751313485114,
+                "hit@10": 0.6260945709281961,
+                "ndcg_linear@1": 0.023642732049036778,
+                "ndcg_linear@5": 0.21440170248700216,
+                "ndcg_linear@10": 0.2875500533246049,
+                "ndcg_exponential@1": 0.023642732049036778,
+                "ndcg_exponential@5": 0.21440170248700216,
+                "ndcg_exponential@10": 0.287550053324605,
+                "precision@1": 0.023642732049036778,
+                "precision@5": 0.08003502626970228,
+                "precision@10": 0.06260945709281962,
+                "recall@1": 0.023642732049036778,
+                "recall@5": 0.4001751313485114,
+                "recall@10": 0.6260945709281961,
+                "f1@1": 0.023642732049036778,
+                "f1@5": 0.1333917104495038,
+                "f1@10": 0.1138353765323993,
+                "map@1": 0.023642732049036778,
+                "map@5": 0.15345884413309982,
+                "map@10": 0.18370097017207349,
+                "mrr@1": 0.023642732049036778,
+                "mrr@5": 0.15345884413309982,
+                "mrr@10": 0.18370097017207349,
+                "num_pred": 1142,
+                "num_data": 1142
+            },
+            "ViDoSeek-doc": {
+                "hit@1": 0.9921190893169878,
+                "hit@5": 0.9956217162872154,
+                "hit@10": 0.9956217162872154,
+                "ndcg_linear@1": 0.6762988908347927,
+                "ndcg_linear@5": 0.8229469185937884,
+                "ndcg_linear@10": 0.8026481335065712,
+                "ndcg_exponential@1": 0.45071303477608193,
+                "ndcg_exponential@5": 0.7032432296216546,
+                "ndcg_exponential@10": 0.7282413355215829,
+                "precision@1": 0.9921190893169878,
+                "precision@5": 0.9084063047285464,
+                "precision@10": 0.8127845884413311,
+                "recall@1": 0.05853541804174077,
+                "recall@5": 0.26730162656135453,
+                "recall@10": 0.4682238375254491,
+                "f1@1": 0.10979939672279285,
+                "f1@5": 0.4046277044815002,
+                "f1@10": 0.5808002886460247,
+                "map@1": 0.9921190893169878,
+                "map@5": 0.8995563339171047,
+                "map@10": 0.7998367054060387,
+                "mrr@1": 0.9921190893169878,
+                "mrr@5": 0.9936514886164624,
+                "mrr@10": 0.9936514886164624,
+                "num_pred": 1142,
+                "num_data": 1142
+            },
+            "MMLongBench-page": {
+                "hit@1": 0.05489260143198091,
+                "hit@5": 0.2649164677804296,
+                "hit@10": 0.3794749403341289,
+                "ndcg_linear@1": 0.05489260143198091,
+                "ndcg_linear@5": 0.13059870824287584,
+                "ndcg_linear@10": 0.16799933893911476,
+                "ndcg_exponential@1": 0.05489260143198091,
+                "ndcg_exponential@5": 0.13059870824287584,
+                "ndcg_exponential@10": 0.16799933893911476,
+                "precision@1": 0.05489260143198091,
+                "precision@5": 0.06014319809069212,
+                "precision@10": 0.048448687350835316,
+                "recall@1": 0.03204057279236277,
+                "recall@5": 0.1906367200818275,
+                "recall@10": 0.2919593135583589,
+                "f1@1": 0.038473062339410787,
+                "f1@5": 0.08654065333484118,
+                "f1@10": 0.07902733863744991,
+                "map@1": 0.05489260143198091,
+                "map@5": 0.09544550517104217,
+                "map@10": 0.11231696321461494,
+                "mrr@1": 0.05489260143198091,
+                "mrr@5": 0.13414876690533015,
+                "mrr@10": 0.14969456756449598,
+                "num_pred": 838,
+                "num_data": 838
+            },
+            "MMLongBench-doc": {
+                "hit@1": 0.6599045346062052,
+                "hit@5": 0.7625298329355609,
+                "hit@10": 0.7971360381861575,
+                "ndcg_linear@1": 0.4594272076372315,
+                "ndcg_linear@5": 0.47888678810724294,
+                "ndcg_linear@10": 0.4572403690751154,
+                "ndcg_exponential@1": 0.3162291169451074,
+                "ndcg_exponential@5": 0.3977515082488963,
+                "ndcg_exponential@10": 0.4053794301498071,
+                "precision@1": 0.6599045346062052,
+                "precision@5": 0.5298329355608592,
+                "precision@10": 0.46264916467780437,
+                "recall@1": 0.0212095103625952,
+                "recall@5": 0.08211773802880712,
+                "recall@10": 0.13971662802131246,
+                "f1@1": 0.04069885876173563,
+                "f1@5": 0.13693384090532812,
+                "f1@10": 0.20204454124738297,
+                "map@1": 0.6599045346062052,
+                "map@5": 0.4958154335719968,
+                "map@10": 0.41324746498970844,
+                "mrr@1": 0.6599045346062052,
+                "mrr@5": 0.7001591089896579,
+                "mrr@10": 0.7046870856536728,
+                "num_pred": 838,
+                "num_data": 838
+            }
+        },
+        "video": {
+            "DiDeMo": {
+                "hit@1": 0.398406374501992,
+                "hit@5": 0.6454183266932271,
+                "hit@10": 0.7440239043824701,
+                "ndcg_linear@1": 0.398406374501992,
+                "ndcg_linear@5": 0.5320235166209488,
+                "ndcg_linear@10": 0.5637579478756262,
+                "ndcg_exponential@1": 0.398406374501992,
+                "ndcg_exponential@5": 0.5320235166209488,
+                "ndcg_exponential@10": 0.5637579478756262,
+                "precision@1": 0.398406374501992,
+                "precision@5": 0.1290836653386454,
+                "precision@10": 0.074402390438247,
+                "recall@1": 0.398406374501992,
+                "recall@5": 0.6454183266932271,
+                "recall@10": 0.7440239043824701,
+                "f1@1": 0.398406374501992,
+                "f1@5": 0.21513944223107576,
+                "f1@10": 0.1352770735240855,
+                "map@1": 0.398406374501992,
+                "map@5": 0.49414010624169985,
+                "map@10": 0.5071551097198508,
+                "mrr@1": 0.398406374501992,
+                "mrr@5": 0.49414010624169985,
+                "mrr@10": 0.5071551097198508,
+                "num_pred": 1004,
+                "num_data": 1004
+            },
+            "MSR-VTT": {
+                "hit@1": 0.46,
+                "hit@5": 0.682,
+                "hit@10": 0.777,
+                "ndcg_linear@1": 0.46,
+                "ndcg_linear@5": 0.577246738740525,
+                "ndcg_linear@10": 0.6082568811617736,
+                "ndcg_exponential@1": 0.46,
+                "ndcg_exponential@5": 0.577246738740525,
+                "ndcg_exponential@10": 0.6082568811617736,
+                "precision@1": 0.46,
+                "precision@5": 0.13639999999999997,
+                "precision@10": 0.0777,
+                "recall@1": 0.46,
+                "recall@5": 0.682,
+                "recall@10": 0.777,
+                "f1@1": 0.46,
+                "f1@5": 0.22733333333333336,
+                "f1@10": 0.14127272727272727,
+                "map@1": 0.46,
+                "map@5": 0.5425333333333333,
+                "map@10": 0.5554920634920635,
+                "mrr@1": 0.46,
+                "mrr@5": 0.5425333333333333,
+                "mrr@10": 0.5554920634920635,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSVD": {
+                "hit@1": 0.6611940298507463,
+                "hit@5": 0.8850746268656716,
+                "hit@10": 0.9432835820895522,
+                "ndcg_linear@1": 0.6611940298507463,
+                "ndcg_linear@5": 0.7846072580329928,
+                "ndcg_linear@10": 0.8036925374517581,
+                "ndcg_exponential@1": 0.6611940298507463,
+                "ndcg_exponential@5": 0.7846072580329928,
+                "ndcg_exponential@10": 0.8036925374517581,
+                "precision@1": 0.6611940298507463,
+                "precision@5": 0.17701492537313432,
+                "precision@10": 0.0943283582089552,
+                "recall@1": 0.6611940298507463,
+                "recall@5": 0.8850746268656716,
+                "recall@10": 0.9432835820895522,
+                "f1@1": 0.6611940298507463,
+                "f1@5": 0.2950248756218906,
+                "f1@10": 0.17150610583446407,
+                "map@1": 0.6611940298507463,
+                "map@5": 0.7507462686567165,
+                "map@10": 0.7587680644397062,
+                "mrr@1": 0.6611940298507463,
+                "mrr@5": 0.7507462686567165,
+                "mrr@10": 0.7587680644397062,
+                "num_pred": 670,
+                "num_data": 670
+            },
+            "VATEX": {
+                "hit@1": 0.40241179097811525,
+                "hit@5": 0.6737382760160786,
+                "hit@10": 0.7753461366681554,
+                "ndcg_linear@1": 0.40241179097811525,
+                "ndcg_linear@5": 0.5467348670052526,
+                "ndcg_linear@10": 0.5796896244320889,
+                "ndcg_exponential@1": 0.40241179097811525,
+                "ndcg_exponential@5": 0.5467348670052526,
+                "ndcg_exponential@10": 0.5796896244320889,
+                "precision@1": 0.40241179097811525,
+                "precision@5": 0.13474765520321572,
+                "precision@10": 0.07753461366681554,
+                "recall@1": 0.40241179097811525,
+                "recall@5": 0.6737382760160786,
+                "recall@10": 0.7753461366681554,
+                "f1@1": 0.40241179097811525,
+                "f1@5": 0.22457942533869293,
+                "f1@10": 0.14097202484875554,
+                "map@1": 0.40241179097811525,
+                "map@5": 0.5044700014887598,
+                "map@10": 0.5181273350489519,
+                "mrr@1": 0.40241179097811525,
+                "mrr@5": 0.5044700014887598,
+                "mrr@10": 0.5181273350489519,
+                "num_pred": 4478,
+                "num_data": 4478
+            },
+            "YouCook2": {
+                "hit@1": 0.16546083674111356,
+                "hit@5": 0.3397294746775716,
+                "hit@10": 0.4510852469329978,
+                "ndcg_linear@1": 0.16546083674111356,
+                "ndcg_linear@5": 0.2557543009200639,
+                "ndcg_linear@10": 0.29139763959836457,
+                "ndcg_exponential@1": 0.16546083674111356,
+                "ndcg_exponential@5": 0.2557543009200639,
+                "ndcg_exponential@10": 0.29139763959836457,
+                "precision@1": 0.16546083674111356,
+                "precision@5": 0.06794589493551431,
+                "precision@10": 0.045108524693299776,
+                "recall@1": 0.16546083674111356,
+                "recall@5": 0.3397294746775716,
+                "recall@10": 0.4510852469329978,
+                "f1@1": 0.16546083674111356,
+                "f1@5": 0.1132431582258572,
+                "f1@10": 0.08201549944236325,
+                "map@1": 0.16546083674111356,
+                "map@5": 0.22801195344447936,
+                "map@10": 0.24250575452997597,
+                "mrr@1": 0.16546083674111356,
+                "mrr@5": 0.22801195344447936,
+                "mrr@10": 0.24250575452997597,
+                "num_pred": 3179,
+                "num_data": 3179
+            },
+            "QVHighlight": {
+                "hit@1": 0.39519852262234534,
+                "hit@5": 0.7857802400738689,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.39519852262234534,
+                "ndcg_linear@5": 0.5970166106855083,
+                "ndcg_linear@10": 0.6657152889063322,
+                "ndcg_exponential@1": 0.39519852262234534,
+                "ndcg_exponential@5": 0.5970166106855083,
+                "ndcg_exponential@10": 0.6657152889063322,
+                "precision@1": 0.39519852262234534,
+                "precision@5": 0.15715604801477376,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.39519852262234534,
+                "recall@5": 0.7857802400738689,
+                "recall@10": 1.0,
+                "f1@1": 0.39519852262234534,
+                "f1@5": 0.2619267466912897,
+                "f1@10": 0.18181818181818185,
+                "map@1": 0.39519852262234534,
+                "map@5": 0.5348107109879964,
+                "map@10": 0.5628358176142109,
+                "mrr@1": 0.39519852262234534,
+                "mrr@5": 0.5348107109879964,
+                "mrr@10": 0.5628358176142109,
+                "num_pred": 1083,
+                "num_data": 1083
+            },
+            "Charades-STA": {
+                "hit@1": 0.1953232462173315,
+                "hit@5": 0.5350756533700137,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.1953232462173315,
+                "ndcg_linear@5": 0.3669548229424312,
+                "ndcg_linear@10": 0.513821217099232,
+                "ndcg_exponential@1": 0.1953232462173315,
+                "ndcg_exponential@5": 0.3669548229424312,
+                "ndcg_exponential@10": 0.513821217099232,
+                "precision@1": 0.1953232462173315,
+                "precision@5": 0.10701513067400273,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.1953232462173315,
+                "recall@5": 0.5350756533700137,
+                "recall@10": 1.0,
+                "f1@1": 0.1953232462173315,
+                "f1@5": 0.17835855112333796,
+                "f1@10": 0.18181818181818185,
+                "map@1": 0.1953232462173315,
+                "map@5": 0.3119440623567171,
+                "map@10": 0.37057433243815635,
+                "mrr@1": 0.1953232462173315,
+                "mrr@5": 0.3119440623567171,
+                "mrr@10": 0.37057433243815635,
+                "num_pred": 727,
+                "num_data": 727
+            },
+            "MomentSeeker": {
+                "hit@1": 0.42,
+                "hit@5": 0.8972222222222223,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.42,
+                "ndcg_linear@5": 0.6748456963769024,
+                "ndcg_linear@10": 0.712776319834093,
+                "ndcg_exponential@1": 0.42,
+                "ndcg_exponential@5": 0.6748456963769024,
+                "ndcg_exponential@10": 0.712776319834093,
+                "precision@1": 0.42,
+                "precision@5": 0.19844444444444448,
+                "precision@10": 0.11261111111111113,
+                "recall@1": 0.39712962962962967,
+                "recall@5": 0.885162037037037,
+                "recall@10": 0.997523148148148,
+                "f1@1": 0.4043068783068783,
+                "f1@5": 0.31906429989763324,
+                "f1@10": 0.1998093573093573,
+                "map@1": 0.42,
+                "map@5": 0.5997861111111111,
+                "map@10": 0.6172150940623162,
+                "mrr@1": 0.42,
+                "mrr@5": 0.607425925925926,
+                "mrr@10": 0.6216602733686066,
+                "num_pred": 1800,
+                "num_data": 1800
+            },
+            "K700": {
+                "hit@1": 0.52,
+                "hit@5": 0.736,
+                "hit@10": 0.81,
+                "ndcg_linear@1": 0.52,
+                "ndcg_linear@5": 0.6375211963083107,
+                "ndcg_linear@10": 0.6617519413591341,
+                "ndcg_exponential@1": 0.52,
+                "ndcg_exponential@5": 0.6375211963083107,
+                "ndcg_exponential@10": 0.6617519413591341,
+                "precision@1": 0.52,
+                "precision@5": 0.1472,
+                "precision@10": 0.081,
+                "recall@1": 0.52,
+                "recall@5": 0.736,
+                "recall@10": 0.81,
+                "f1@1": 0.52,
+                "f1@5": 0.24533333333333338,
+                "f1@10": 0.14727272727272725,
+                "map@1": 0.52,
+                "map@5": 0.6045833333333333,
+                "map@10": 0.6147547619047619,
+                "mrr@1": 0.52,
+                "mrr@5": 0.6045833333333333,
+                "mrr@10": 0.6147547619047619,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "SmthSmthV2": {
+                "hit@1": 0.589,
+                "hit@5": 0.821,
+                "hit@10": 0.893,
+                "ndcg_linear@1": 0.589,
+                "ndcg_linear@5": 0.7154946142944708,
+                "ndcg_linear@10": 0.7390605810798436,
+                "ndcg_exponential@1": 0.589,
+                "ndcg_exponential@5": 0.7154946142944708,
+                "ndcg_exponential@10": 0.7390605810798436,
+                "precision@1": 0.589,
+                "precision@5": 0.16419999999999998,
+                "precision@10": 0.08930000000000002,
+                "recall@1": 0.589,
+                "recall@5": 0.821,
+                "recall@10": 0.893,
+                "f1@1": 0.589,
+                "f1@5": 0.27366666666666667,
+                "f1@10": 0.16236363636363638,
+                "map@1": 0.589,
+                "map@5": 0.6800999999999999,
+                "map@10": 0.6899845238095238,
+                "mrr@1": 0.589,
+                "mrr@5": 0.6800999999999999,
+                "mrr@10": 0.6899845238095238,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "HMDB51": {
+                "hit@1": 0.555,
+                "hit@5": 0.765,
+                "hit@10": 0.835,
+                "ndcg_linear@1": 0.555,
+                "ndcg_linear@5": 0.6674635098696148,
+                "ndcg_linear@10": 0.6904162436303634,
+                "ndcg_exponential@1": 0.555,
+                "ndcg_exponential@5": 0.6674635098696148,
+                "ndcg_exponential@10": 0.6904162436303634,
+                "precision@1": 0.555,
+                "precision@5": 0.153,
+                "precision@10": 0.0835,
+                "recall@1": 0.555,
+                "recall@5": 0.765,
+                "recall@10": 0.835,
+                "f1@1": 0.555,
+                "f1@5": 0.25500000000000006,
+                "f1@10": 0.15181818181818182,
+                "map@1": 0.555,
+                "map@5": 0.6349333333333332,
+                "map@10": 0.6446035714285714,
+                "mrr@1": 0.555,
+                "mrr@5": 0.6349333333333332,
+                "mrr@10": 0.6446035714285714,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "UCF101": {
+                "hit@1": 0.714,
+                "hit@5": 0.9,
+                "hit@10": 0.925,
+                "ndcg_linear@1": 0.714,
+                "ndcg_linear@5": 0.8167259122259789,
+                "ndcg_linear@10": 0.824653618618315,
+                "ndcg_exponential@1": 0.714,
+                "ndcg_exponential@5": 0.8167259122259789,
+                "ndcg_exponential@10": 0.824653618618315,
+                "precision@1": 0.714,
+                "precision@5": 0.18000000000000002,
+                "precision@10": 0.0925,
+                "recall@1": 0.714,
+                "recall@5": 0.9,
+                "recall@10": 0.925,
+                "f1@1": 0.714,
+                "f1@5": 0.3,
+                "f1@10": 0.16818181818181818,
+                "map@1": 0.714,
+                "map@5": 0.7886166666666666,
+                "map@10": 0.7918007936507936,
+                "mrr@1": 0.714,
+                "mrr@5": 0.7886166666666666,
+                "mrr@10": 0.7918007936507936,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Breakfast": {
+                "hit@1": 0.302540415704388,
+                "hit@5": 0.7621247113163973,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.302540415704388,
+                "ndcg_linear@5": 0.5350005551553058,
+                "ndcg_linear@10": 0.6133736879361149,
+                "ndcg_exponential@1": 0.302540415704388,
+                "ndcg_exponential@5": 0.5350005551553058,
+                "ndcg_exponential@10": 0.6133736879361149,
+                "precision@1": 0.302540415704388,
+                "precision@5": 0.15242494226327943,
+                "precision@10": 0.1,
+                "recall@1": 0.302540415704388,
+                "recall@5": 0.7621247113163973,
+                "recall@10": 1.0,
+                "f1@1": 0.302540415704388,
+                "f1@5": 0.25404157043879916,
+                "f1@10": 0.1818181818181818,
+                "map@1": 0.302540415704388,
+                "map@5": 0.4606620477290223,
+                "map@10": 0.49383316837127456,
+                "mrr@1": 0.302540415704388,
+                "mrr@5": 0.4606620477290223,
+                "mrr@10": 0.49383316837127456,
+                "num_pred": 433,
+                "num_data": 433
+            },
+            "MVBench": {
+                "hit@1": 0.5075,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.5075,
+                "ndcg_linear@5": 0.7815510541301905,
+                "ndcg_linear@10": 0.7815510541301905,
+                "ndcg_exponential@1": 0.5075,
+                "ndcg_exponential@5": 0.7815510541301905,
+                "ndcg_exponential@10": 0.7815510541301905,
+                "precision@1": 0.5075,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.5075,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.5075,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.5075,
+                "map@5": 0.7074625000000001,
+                "map@10": 0.7074625000000001,
+                "mrr@1": 0.5075,
+                "mrr@5": 0.7074625000000001,
+                "mrr@10": 0.7074625000000001,
+                "num_pred": 4000,
+                "num_data": 4000
+            },
+            "Video-MME": {
+                "hit@1": 0.4462962962962963,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.4462962962962963,
+                "ndcg_linear@5": 0.744369697755487,
+                "ndcg_linear@10": 0.744369697755487,
+                "ndcg_exponential@1": 0.4462962962962963,
+                "ndcg_exponential@5": 0.744369697755487,
+                "ndcg_exponential@10": 0.744369697755487,
+                "precision@1": 0.4462962962962963,
+                "precision@5": 0.19999999999999996,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.4462962962962963,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.4462962962962963,
+                "f1@5": 0.3333333333333334,
+                "f1@10": 0.18181818181818188,
+                "map@1": 0.4462962962962963,
+                "map@5": 0.6585493827160493,
+                "map@10": 0.6585493827160493,
+                "mrr@1": 0.4462962962962963,
+                "mrr@5": 0.6585493827160493,
+                "mrr@10": 0.6585493827160493,
+                "num_pred": 2700,
+                "num_data": 2700
+            },
+            "NExTQA": {
+                "hit@1": 0.5726296123306865,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.5726296123306865,
+                "ndcg_linear@5": 0.799026160227503,
+                "ndcg_linear@10": 0.799026160227503,
+                "ndcg_exponential@1": 0.5726296123306865,
+                "ndcg_exponential@5": 0.799026160227503,
+                "ndcg_exponential@10": 0.799026160227503,
+                "precision@1": 0.5726296123306865,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.5726296123306865,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.5726296123306865,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.1818181818181818,
+                "map@1": 0.5726296123306865,
+                "map@5": 0.732224038611241,
+                "map@10": 0.732224038611241,
+                "mrr@1": 0.5726296123306865,
+                "mrr@5": 0.7320488868130158,
+                "mrr@10": 0.7320488868130158,
+                "num_pred": 8564,
+                "num_data": 8564
+            },
+            "EgoSchema": {
+                "hit@1": 0.538,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.538,
+                "ndcg_linear@5": 0.7801130497021113,
+                "ndcg_linear@10": 0.7801130497021113,
+                "ndcg_exponential@1": 0.538,
+                "ndcg_exponential@5": 0.7801130497021113,
+                "ndcg_exponential@10": 0.7801130497021113,
+                "precision@1": 0.538,
+                "precision@5": 0.2,
+                "precision@10": 0.1,
+                "recall@1": 0.538,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.538,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.538,
+                "map@5": 0.7072666666666666,
+                "map@10": 0.7072666666666666,
+                "mrr@1": 0.538,
+                "mrr@5": 0.7072666666666666,
+                "mrr@10": 0.7072666666666666,
+                "num_pred": 500,
+                "num_data": 500
+            },
+            "ActivityNetQA": {
+                "hit@1": 0.718,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.718,
+                "ndcg_linear@5": 0.895922190507151,
+                "ndcg_linear@10": 0.895922190507151,
+                "ndcg_exponential@1": 0.718,
+                "ndcg_exponential@5": 0.895922190507151,
+                "ndcg_exponential@10": 0.895922190507151,
+                "precision@1": 0.718,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.718,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.718,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.718,
+                "map@5": 0.859,
+                "map@10": 0.859,
+                "mrr@1": 0.718,
+                "mrr@5": 0.859,
+                "mrr@10": 0.859,
+                "num_pred": 1000,
+                "num_data": 1000
+            }
+        }
+    }
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:091aa7594dc2fcfbfa06b9e3c22a5f0562ac14f30375c13af7309407a0e67b8a
+size 11420371

tokenizer_config.json

@@ -0,0 +1,145 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "processor_class": "Qwen2VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}