Update README.md

README.md

@@ -30,55 +30,51 @@ datasets:
 - sbu_captions
 - visual_genome
 - ChristophSchuhmann/MS_COCO_2017_URL_TEXT
+- Ziyang/yfcc15m
 ---
 
 <h1 align="center">UForm</h1>
 <h3 align="center">
-Multi-Modal Inference Library<br/>
-For Semantic Search Applications<br/>
+Pocket-Sized Multimodal AI<br/>
+For Content Understanding and Generation<br/>
+In Python, JavaScript, and Swift<br/>
 </h3>
 
 ---
 
-UForm is a Multi-Modal Modal Inference package, designed to encode Multi-Lingual Texts, Images, and, soon, Audio, Video, and Documents, into a shared vector space!
+The `uform3-image-text-multilingual-base` UForm model is a tiny vision and multilingual language encoder, covering __21 languages__, mapping them into a shared vector space.
+This model produces up to __256-dimensional embeddings__ and is made of:
 
-This is model card of the __Multilingual model__ (21 languages) with:
+* Text encoder: 12-layer BERT for up to 50 input tokens.
+* Visual encoder: ViT-B/16 for images of 224 x 224 resolution.
 
-* 12 layers BERT (8 layers for unimodal encoding and rest layers for multimodal encoding)
-* ViT-B/16 (image resolution is 224x224)
-
-The model was trained on balanced multilingual dataset.
-
-If you need English model, check [this](https://huggingface.co/unum-cloud/uform-vl-english).
+Unlike most CLIP-like multomodal models, this model shares 4 layers between the text and visual encoder to allow for more data- and parameter-efficient training.
+Also unlike most models, UForm provides checkpoints compatible with PyTorch, ONNX, and CoreML, covering the absolute majority of AI-capable devices, with pre-quantized weights and inference code.
+If you need a larger, more accurate, or multilingual model, check our [HuggingFace Hub](https://huggingface.co/unum-cloud/).
+For more details on running the model, check out the [UForm GitHub repository](https://github.com/unum-cloud/uform/).
 
 ## Evaluation
 
 For all evaluations, the multimodal part was used unless otherwise stated.
 
-**Monolingual**
+### Monolingual
 
 | Dataset |  Recall@1 |  Recall@5 | Recall@10 |
 | :-------- | ------: | --------: | --------: |
 | Zero-Shot Flickr | 0.558 | 0.813 | 0.874 |
-| MS-COCO (train split was in training data) | 0.401 | 0.680 | 0.781 |
-
-**Multilingual**
+| MS-COCO ¹ | 0.401 | 0.680 | 0.781 |
 
-[XTD-10](https://github.com/adobe-research/Cross-lingual-Test-Dataset-XTD10)
+> ¹ It's important to note, that the MS-COCO train split was present in the training data.
 
- Metric is recall@10
+### Multilingual
 
+Recall@10 on the [XTD-10](https://github.com/adobe-research/Cross-lingual-Test-Dataset-XTD10) dataset:
 
 |  English |   German |  Spanish |   French |  Italian |  Russian | Japanese |   Korean |  Turkish |  Chinese | Polish |
 | -------: | -------: | -------: | -------: | -------: | -------: | -------: | -------: | -------: | -------: | ------:|
 |     96.1 |     93.5 |     95.7 |     94.1 |     94.4 |     90.4 |     90.2 |     91.3 |     95.2 |     93.8 |   95.8 |
 
-
-[COCO-SM](https://github.com/kimihailv/coco-sm/tree/main)
-
-For this evaluation only unimodal part was used.
-
-Recall
+Recall@1, Recall@5, and Recall@10 on the [COCO-SM](https://github.com/kimihailv/coco-sm/tree/main) dataset:
 
 | Target Language       | OpenCLIP @ 1 | UForm @ 1     | OpenCLIP @ 5 | UForm @ 5     | OpenCLIP @ 10 | UForm @ 10     | Speakers |
 | :-------------------- | -----------: | ------------: | -----------: | -------------:| ------------: | --------------:| -------: |
@@ -109,17 +105,18 @@ Recall
 | Microsoft Translator  |     27.2±6.4 |  **31.4±3.6** |     50.8±9.8 |  **57.7±4.7** |     61.4±10.6 |   **68.9±4.6** |        - |
 | Meta NLLB             |     24.9±6.7 |  **32.4±3.5** |    47.5±10.3 |  **58.9±4.5** |     58.2±11.2 |   **70.2±4.3** |        - |
 
-NDCG@20
+For a deeper comparison of output ranking check the following table for the Normalized Discounted Cumulative Gains for the first 20 results - NDCG@20:
 
 |               |     Arabic |     Armenian |     Chinese |     French |     German |     Hebrew |     Hindi |     Indonesian |     Italian |     Japanese |     Korean |     Persian |     Polish |     Portuguese |     Russian |     Spanish |     Thai |     Turkish |     Ukranian |     Vietnamese |   Mean (all) | Mean (Google Translate) | Mean(Microsoft Translator) | Mean(NLLB)
 | :------------ | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: | ----: |
 | OpenCLIP NDCG | 0.639 | 0.204 | 0.731 | 0.823 | 0.806 | 0.657 | 0.616 | 0.733 | 0.811 | 0.737 | 0.686 | 0.667 | 0.764 | 0.832 | 0.777 | 0.849 | 0.606 | 0.701 | 0.704 | 0.697 | 0.716 ± 0.149 | 0.732 ± 0.145 | 0.730 ± 0.149 | 0.686 ± 0.158
 | UForm NDCG    | 0.868 | 0.691 | 0.880 | 0.932 | 0.927 | 0.791 | 0.879 | 0.870 | 0.930 | 0.885 | 0.869 | 0.831 | 0.897 | 0.897 | 0.906 | 0.939 | 0.822 | 0.898 | 0.851 | 0.818 | 0.875 ± 0.064 | 0.869 ± 0.063 | 0.869 ± 0.066 | 0.888 ± 0.064
 
+
 ## Installation
 
 ```bash
-pip install uform[torch]
+pip install "uform[torch,onnx]"
 ```
 
 ## Usage
@@ -127,82 +124,31 @@ pip install uform[torch]
 To load the model:
 
 ```python
-import uform
-
-model, processor = uform.get_model('unum-cloud/uform-vl-multilingual-v2')
-```
-
-To encode data:
+from uform import get_model, Modality
 
-```python
+import requests
+from io import BytesIO
 from PIL import Image
 
-text = 'a small red panda in a zoo'
-image = Image.open('red_panda.jpg')
-
-image_data = processor.preprocess_image(image)
-text_data = processor.preprocess_text(text)
+model_name = 'unum-cloud/uform3-image-text-multilingual-base'
+modalities = [Modality.TEXT_ENCODER, Modality.IMAGE_ENCODER]
+processors, models = get_model(model_name, modalities=modalities)
 
-image_features, image_embedding = model.encode_image(image_data, return_features=True)
-text_features, text_embedding = model.encode_text(text_data, return_features=True)
-joint_embedding = model.encode_multimodal(image=image_data, text=text_data)
+model_text = models[Modality.TEXT_ENCODER]
+model_image = models[Modality.IMAGE_ENCODER]
+processor_text = processors[Modality.TEXT_ENCODER]
+processor_image = processors[Modality.IMAGE_ENCODER]
 ```
 
-To get features:
+To encode the content:
 
 ```python
-image_features, image_embedding = model.encode_image(image_data, return_features=True)
-text_features, text_embedding = model.encode_text(text_data, return_features=True)
+text = 'a cityscape bathed in the warm glow of the sun, with varied architecture and a towering, snow-capped mountain rising majestically in the background'
+image_url = 'https://media-cdn.tripadvisor.com/media/photo-s/1b/28/6b/53/lovely-armenia.jpg'
+image_url = Image.open(BytesIO(requests.get(image_url).content))
+
+image_data = processor_image(image)
+text_data = processor_text(text)
+image_features, image_embedding = model_image.encode(image_data, return_features=True)
+text_features, text_embedding = model_text.encode(text_data, return_features=True)
 ```
-
-These features can later be used to produce joint multimodal encodings faster, as the first layers of the transformer can be skipped:
-
-```python
-joint_embedding = model.encode_multimodal(
-    image_features=image_features,
-    text_features=text_features,
-    attention_mask=text_data['attention_mask']
-)
-```
-
-There are two options to calculate semantic compatibility between an image and a text: [Cosine Similarity](#cosine-similarity) and [Matching Score](#matching-score).
-
-### Cosine Similarity
-
-```python
-import torch.nn.functional as F
-
-similarity = F.cosine_similarity(image_embedding, text_embedding)
-```
-
-The `similarity` will belong to the `[-1, 1]` range, `1` meaning the absolute match.
-
-__Pros__:
-
-- Computationally cheap.
-- Only unimodal embeddings are required, unimodal encoding is faster than joint encoding.
-- Suitable for retrieval in large collections.
-
-__Cons__:
-
-- Takes into account only coarse-grained features.
-
-
-### Matching Score 
-
-Unlike cosine similarity, unimodal embedding are not enough.
-Joint embedding will be needed and the resulting `score` will belong to the `[0, 1]` range, `1` meaning the absolute match.
-
-```python
-score = model.get_matching_scores(joint_embedding)
-```
-
-__Pros__:
-
-- Joint embedding captures fine-grained features.
-- Suitable for re-ranking – sorting retrieval result.
-
-__Cons__:
-
-- Resource-intensive.
-- Not suitable for retrieval in large collections.