Upload folder using huggingface_hub

README.md

@@ -0,0 +1,208 @@
+---
+license: apache-2.0
+language:
+- en
+- ar
+- hy
+- zh
+- fr
+- de
+- he
+- hi
+- id
+- it
+- ja
+- ko
+- fa
+- pl
+- pt
+- ru
+- es
+- th
+- tr
+- uk
+- vi
+pipeline_tag: feature-extraction
+tags:
+- clip
+- vision
+datasets:
+- sbu_captions
+- visual_genome
+- ChristophSchuhmann/MS_COCO_2017_URL_TEXT
+---
+
+<h1 align="center">UForm</h1>
+<h3 align="center">
+Multi-Modal Inference Library<br/>
+For Semantic Search Applications<br/>
+</h3>
+
+---
+
+UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
+
+This is model card of the __Multilingual model__ (21 languages) with:
+
+* 12 layers BERT (8 layers for unimodal encoding and rest layers for multimodal encoding)
+* ViT-B/16 (image resolution is 224x224)
+
+The model was trained on balanced multilingual dataset.
+
+If you need English model, check [this](https://huggingface.co/unum-cloud/uform-vl-english).
+
+## Evaluation
+
+For all evaluations, the multimodal part was used unless otherwise stated.
+
+**Monolingual**
+
+| Dataset |  Recall@1 |  Recall@5 | Recall@10 |
+| :-------- | ------: | --------: | --------: |
+| Zero-Shot Flickr | 0.558 | 0.813 | 0.874 |
+| MS-COCO (train split was in training data) | 0.401 | 0.680 | 0.781 |
+
+**Multilingual**
+
+[XTD-10](https://github.com/adobe-research/Cross-lingual-Test-Dataset-XTD10)
+
+ Metric is recall@10
+
+
+|  English |   German |  Spanish |   French |  Italian |  Russian | Japanese |   Korean |  Turkish |  Chinese | Polish |
+| -------: | -------: | -------: | -------: | -------: | -------: | -------: | -------: | -------: | -------: | ------:|
+|     96.1 |     93.5 |     95.7 |     94.1 |     94.4 |     90.4 |     90.2 |     91.3 |     95.2 |     93.8 |   95.8 |
+
+
+[COCO-SM](https://github.com/kimihailv/coco-sm/tree/main)
+
+For this evaluation only unimodal part was used.
+
+Recall
+
+| Target Language       | OpenCLIP @ 1 | UForm @ 1     | OpenCLIP @ 5 | UForm @ 5     | OpenCLIP @ 10 | UForm @ 10     | Speakers |
+| :-------------------- | -----------: | ------------: | -----------: | -------------:| ------------: | --------------:| -------: |
+| Arabic             |         22.7 |      **31.7** |         44.9 |      **57.8** |          55.8 |       **69.2** |    274 M |
+| Armenian           |          5.6 |      **22.0** |         14.3 |      **44.7** |          20.2 |       **56.0** |      4 M |
+| Chinese            |         27.3 |      **32.2** |         51.3 |      **59.0** |          62.1 |       **70.5** |  1'118 M |
+| English            |     **37.8** |          37.7 |         63.5 |      **65.0** |          73.5 |       **75.9** |  1'452 M |
+| French             |         31.3 |      **35.4** |         56.5 |      **62.6** |          67.4 |       **73.3** |    274 M |
+| German             |         31.7 |      **35.1** |         56.9 |      **62.2** |          67.4 |       **73.3** |    134 M |
+| Hebrew             |         23.7 |      **26.7** |         46.3 |      **51.8** |          57.0 |       **63.5** |      9 M |
+| Hindi              |         20.7 |      **31.3** |         42.5 |      **57.9** |          53.7 |       **69.6** |    602 M |
+| Indonesian         |         26.9 |      **30.7** |         51.4 |      **57.0** |          62.7 |       **68.6** |    199 M |
+| Italian            |         31.3 |      **34.9** |         56.7 |      **62.1** |          67.1 |       **73.1** |     67 M |
+| Japanese           |         27.4 |      **32.6** |         51.5 |      **59.2** |          62.6 |       **70.6** |    125 M |
+| Korean             |         24.4 |      **31.5** |         48.1 |      **57.8** |          59.2 |       **69.2** |     81 M |
+| Persian            |         24.0 |      **28.8** |         47.0 |      **54.6** |          57.8 |       **66.2** |     77 M |
+| Polish             |         29.2 |      **33.6** |         53.9 |      **60.1** |          64.7 |       **71.3** |     41 M |
+| Portuguese         |         31.6 |      **32.7** |         57.1 |      **59.6** |          67.9 |       **71.0** |    257 M |
+| Russian            |         29.9 |      **33.9** |         54.8 |      **60.9** |          65.8 |       **72.0** |    258 M |
+| Spanish            |         32.6 |      **35.6** |         58.0 |      **62.8** |          68.8 |       **73.7** |    548 M |
+| Thai               |         21.5 |      **28.7** |         43.0 |      **54.6** |          53.7 |       **66.0** |     61 M |
+| Turkish            |         25.5 |      **33.0** |         49.1 |      **59.6** |          60.3 |       **70.8** |     88 M |
+| Ukranian           |         26.0 |      **30.6** |         49.9 |      **56.7** |          60.9 |       **68.1** |     41 M |
+| Vietnamese         |         25.4 |      **28.3** |         49.2 |      **53.9** |          60.3 |       **65.5** |     85 M |
+|                       |              |               |              |               |               |                |          |
+| Mean                  |     26.5±6.4 |  **31.8±3.5** |     49.8±9.8 |  **58.1±4.5** |     60.4±10.6 |   **69.4±4.3** |        - |
+| Google Translate      |     27.4±6.3 |  **31.5±3.5** |     51.1±9.5 |  **57.8±4.4** |     61.7±10.3 |   **69.1±4.3** |        - |
+| Microsoft Translator  |     27.2±6.4 |  **31.4±3.6** |     50.8±9.8 |  **57.7±4.7** |     61.4±10.6 |   **68.9±4.6** |        - |
+| Meta NLLB             |     24.9±6.7 |  **32.4±3.5** |    47.5±10.3 |  **58.9±4.5** |     58.2±11.2 |   **70.2±4.3** |        - |
+
+NDCG@20
+
+|               |     Arabic |     Armenian |     Chinese |     French |     German |     Hebrew |     Hindi |     Indonesian |     Italian |     Japanese |     Korean |     Persian |     Polish |     Portuguese |     Russian |     Spanish |     Thai |     Turkish |     Ukranian |     Vietnamese |   Mean (all) | Mean (Google Translate) | Mean(Microsoft Translator) | Mean(NLLB)
+| :------------ | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: |
+| OpenCLIP NDCG | 0.639 | 0.204 | 0.731 | 0.823 | 0.806 | 0.657 | 0.616 | 0.733 | 0.811 | 0.737 | 0.686 | 0.667 | 0.764 | 0.832 | 0.777 | 0.849 | 0.606 | 0.701 | 0.704 | 0.697 | 0.716 ± 0.149 | 0.732 ± 0.145 | 0.730 ± 0.149 | 0.686 ± 0.158
+| UForm NDCG    | 0.868 | 0.691 | 0.880 | 0.932 | 0.927 | 0.791 | 0.879 | 0.870 | 0.930 | 0.885 | 0.869 | 0.831 | 0.897 | 0.897 | 0.906 | 0.939 | 0.822 | 0.898 | 0.851 | 0.818 | 0.875 ± 0.064 | 0.869 ± 0.063 | 0.869 ± 0.066 | 0.888 ± 0.064
+
+## Installation
+
+```bash
+pip install uform[torch]
+```
+
+## Usage
+
+To load the model:
+
+```python
+import uform
+
+model, processor = uform.get_model('unum-cloud/uform-vl-multilingual-v2')
+```
+
+To encode data:
+
+```python
+from PIL import Image
+
+text = 'a small red panda in a zoo'
+image = Image.open('red_panda.jpg')
+
+image_data = processor.preprocess_image(image)
+text_data = processor.preprocess_text(text)
+
+image_features, image_embedding = model.encode_image(image_data, return_features=True)
+text_features, text_embedding = model.encode_text(text_data, return_features=True)
+joint_embedding = model.encode_multimodal(image=image_data, text=text_data)
+```
+
+To get features:
+
+```python
+image_features, image_embedding = model.encode_image(image_data, return_features=True)
+text_features, text_embedding = model.encode_text(text_data, return_features=True)
+```
+
+These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
+
+```python
+joint_embedding = model.encode_multimodal(
+    image_features=image_features,
+    text_features=text_features,
+    attention_mask=text_data['attention_mask']
+)
+```
+
+There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
+
+### Cosine Similarity
+
+```python
+import torch.nn.functional as F
+
+similarity = F.cosine_similarity(image_embedding, text_embedding)
+```
+
+The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
+
+__Pros__:
+
+- Computationally cheap.
+- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
+- Suitable for retrieval in large collections.
+
+__Cons__:
+
+- Takes into account only coarse-grained features.
+
+
+### Matching Score 
+
+Unlike cosine similarity, unimodal embedding are not enough.
+Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
+
+```python
+score = model.get_matching_scores(joint_embedding)
+```
+
+__Pros__:
+
+- Joint embedding captures fine-grained features.
+- Suitable for re-ranking – sorting retrieval result.
+
+__Cons__:
+
+- Resource-intensive.
+- Not suitable for retrieval in large collections.

config.json

@@ -0,0 +1,42 @@
+{
+    "text_encoder": {
+        "tokenizer_class": "bert",
+        "model_type": "bert",
+        "dim": 384,
+        "context_dim": 768,
+        "vocab_size": 250037,
+        "padding_idx": 1,
+        "num_layers": 12,
+        "num_heads": 12,
+        "embedding_dim": 256,
+        "multimodal_layers_ids": [
+            8,
+            9,
+            10,
+            11
+        ],
+        "head_one_neuron": false,
+        "pooling": "mean",
+        "max_position_embeddings": 50,
+        "dropout_prob": 0.1
+    },
+    "image_encoder": {
+        "normalization_means": [
+            0.48145466,
+            0.4578275,
+            0.40821073
+        ],
+        "normalization_deviations": [
+            0.26862954,
+            0.26130258,
+            0.27577711
+        ],
+        "dim": 768,
+        "patch_size": 16,
+        "image_size": 224,
+        "num_layers": 12,
+        "num_heads": 12,
+        "embedding_dim": 256,
+        "pooling": "cls"
+    }
+}

image_encoder.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:101d2a485d7f1fb9b13bb2e9b21017d41e0484c09d7252fa48cd45663bf6a7a0
+size 111239

image_encoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b24c24003fc3acf7635ad22f5c7114e2332dcdc7d0bc54ca0a037c777240646
+size 344067136

image_encoder.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "690CB1E0-62DD-45EA-948E-FBBD7B246BE6": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        },
+        "83ED2C72-7659-440D-A7C9-61793559340B": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        }
+    },
+    "rootModelIdentifier": "690CB1E0-62DD-45EA-948E-FBBD7B246BE6"
+}

image_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b764843eeaa20a981f5f1baf19f4fedf33ee0385483fc0d30e15d5aa6ea4912e
+size 87267684

image_encoder.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02b7d4306999f49a70ddfe60e69932428a9c81709bef702f99d9c4380f987645
+size 172100742

image_encoder.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb3c5dc0ecabbafcaab8011112b07ef01d39e006cae25de0b7cc60dc0b42213c
+size 172047072

text_encoder.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b388380d8fdcd0271c9b851c22348453c36463cf43aa3d656b7fa9aa6ea52c42
+size 74060

text_encoder.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:064e0da61cc46b74a9a5f799c529fcbdb00cb19b6603e152b6ede7e3bdbed017
+size 441322432

text_encoder.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "09A10422-686F-432D-B684-1B0F05B38FC0": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        },
+        "0C92F7F7-B285-45D6-9DF1-DCC71E38166D": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        }
+    },
+    "rootModelIdentifier": "0C92F7F7-B285-45D6-9DF1-DCC71E38166D"
+}

text_encoder.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4dfc9d14b13c871a55386fdfc9d787446535dcb013c9ccf8c9db0323db3febf
+size 110689743

text_encoder.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:373485cd65cc62c3e41bf623b75171981fe1160e13f84d5f05499e8e2c34e63f
+size 240262678

text_encoder.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:400b19de2ca7380307010f006f2823cd953fa4d63d90ec5d2e572beb2d5dc512
+size 240204124