Update README.md
Browse files
README.md
CHANGED
|
@@ -1,4 +1,12 @@
|
|
| 1 |
# TΒ³: Test-Time Model Merging for Medical Vision-Language Models
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |

|
| 4 |
*Figure 1: Dynamic test-time merging workflow of TΒ³*
|
|
@@ -22,13 +30,14 @@ Official implementation of **TΒ³: Test-Time Model Merging in Vision-Language Mod
|
|
| 22 |
- [Folder Structure](#folder-structure)
|
| 23 |
- [Reproducing Results](#reproducing-results)
|
| 24 |
- [Pretrained Weights](#pretrained-weights)
|
|
|
|
| 25 |
- [Citation](#citation)
|
| 26 |
|
| 27 |
## Installation
|
| 28 |
|
| 29 |
1. Clone repository:
|
| 30 |
```bash
|
| 31 |
-
git clone https://github.com/
|
| 32 |
cd T3
|
| 33 |
```
|
| 34 |
|
|
@@ -91,12 +100,14 @@ Empirically, \(I(x)\) correlates positively with \(R(x)\), but better distinguis
|
|
| 91 |
---
|
| 92 |
|
| 93 |
## Folder Structure
|
| 94 |
-
|
| 95 |
```
|
| 96 |
T3/
|
| 97 |
βββ clip/ # CLIP model adaptations
|
| 98 |
βββ data/ # Data Utilities
|
| 99 |
βββ utils/ # Helper functions
|
|
|
|
|
|
|
| 100 |
βββ baselines.py # Comparison methods
|
| 101 |
βββ t_cube.py # Core TΒ³ implementation
|
| 102 |
βββ BetaMixture.py # Auxiliary models
|
|
@@ -109,7 +120,7 @@ T3/
|
|
| 109 |
|
| 110 |
To reproduce the results from the paper, you can run the `t_cube.py` script. This script handles the evaluation of TΒ³ and its baselines across multiple datasets and severity levels. Additional baselines are available in `baselines.py`.
|
| 111 |
|
| 112 |
-
To understand the script better
|
| 113 |
- Refer to the `compute_samplewise_tcube_weights` and `compute_samplewise_tcube_weights_MI` functions for entropy (DaWiN baseline) and Our mutual information-based merging.
|
| 114 |
- Check the `evaluate_on_test_set` function for how datasets and severities are processed.
|
| 115 |
- Explore the `evaluate_tcube` function for the merging and evaluation logic.
|
|
@@ -126,12 +137,38 @@ We provide pretrained weights for the following models:
|
|
| 126 |
- Cell Microscopy
|
| 127 |
- Retinal OCT
|
| 128 |
|
| 129 |
-
If you would like access to these weights, please contact us directly at [Raza Imam](mailto:[email protected]).
|
|
|
|
| 130 |
|
| 131 |
---
|
| 132 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
## License
|
| 134 |
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
| 135 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
## Contact
|
| 137 |
-
For questions or collaborations, contact [Raza Imam](mailto:[email protected]).
|
|
|
|
| 1 |
# TΒ³: Test-Time Model Merging for Medical Vision-Language Models
|
| 2 |
+
[Raza Imam](https://razaimam45.github.io/), Hu Wang, Dwarikanath Mahapatra, Mohammad Yaqub \
|
| 3 |
+
Mohamed bin Zayed University of Artificial Intelligence
|
| 4 |
+
|
| 5 |
+
[](LICENSE)
|
| 6 |
+
[](https://arxiv.org/abs/2510.27265)
|
| 7 |
+
[](https://huggingface.co/razaimam45/TCube_Merging)
|
| 8 |
+
|
| 9 |
+
This repository provides the official PyTorch implementation of our TΒ³ Medical Model-Merging paper:
|
| 10 |
|
| 11 |

|
| 12 |
*Figure 1: Dynamic test-time merging workflow of TΒ³*
|
|
|
|
| 30 |
- [Folder Structure](#folder-structure)
|
| 31 |
- [Reproducing Results](#reproducing-results)
|
| 32 |
- [Pretrained Weights](#pretrained-weights)
|
| 33 |
+
- [Datasets](#datasets)
|
| 34 |
- [Citation](#citation)
|
| 35 |
|
| 36 |
## Installation
|
| 37 |
|
| 38 |
1. Clone repository:
|
| 39 |
```bash
|
| 40 |
+
git clone https://github.com/Razaimam45/TCube.git
|
| 41 |
cd T3
|
| 42 |
```
|
| 43 |
|
|
|
|
| 100 |
---
|
| 101 |
|
| 102 |
## Folder Structure
|
| 103 |
+
Do check our [HuggingFace page](https://huggingface.co/razaimam45/TCube_Merging) for Expert Models and Evaluation Datasets.
|
| 104 |
```
|
| 105 |
T3/
|
| 106 |
βββ clip/ # CLIP model adaptations
|
| 107 |
βββ data/ # Data Utilities
|
| 108 |
βββ utils/ # Helper functions
|
| 109 |
+
βββ models/ # Put your finetuned models HERE
|
| 110 |
+
βββ dataset/ # Put your medimeta/medmnist-c eval data HERE
|
| 111 |
βββ baselines.py # Comparison methods
|
| 112 |
βββ t_cube.py # Core TΒ³ implementation
|
| 113 |
βββ BetaMixture.py # Auxiliary models
|
|
|
|
| 120 |
|
| 121 |
To reproduce the results from the paper, you can run the `t_cube.py` script. This script handles the evaluation of TΒ³ and its baselines across multiple datasets and severity levels. Additional baselines are available in `baselines.py`.
|
| 122 |
|
| 123 |
+
To understand the script better; in `t_cube.py`:
|
| 124 |
- Refer to the `compute_samplewise_tcube_weights` and `compute_samplewise_tcube_weights_MI` functions for entropy (DaWiN baseline) and Our mutual information-based merging.
|
| 125 |
- Check the `evaluate_on_test_set` function for how datasets and severities are processed.
|
| 126 |
- Explore the `evaluate_tcube` function for the merging and evaluation logic.
|
|
|
|
| 137 |
- Cell Microscopy
|
| 138 |
- Retinal OCT
|
| 139 |
|
| 140 |
+
<!-- If you would like access to these weights, please contact us directly at [Raza Imam](mailto:[email protected]). -->
|
| 141 |
+
If you would like to access these weights, please find them at model card at [https://huggingface.co/razaimam45/TCube_Merging](https://huggingface.co/razaimam45/TCube_Merging) under `models/finetuned` subfolder.
|
| 142 |
|
| 143 |
---
|
| 144 |
|
| 145 |
+
## Datasets
|
| 146 |
+
|
| 147 |
+
We provided `Breast Imaging` evaluation sets on [HuggingFace page](https://huggingface.co/razaimam45/TCube_Merging). Please download from there.
|
| 148 |
+
|
| 149 |
+
If you need to run multiple modalities datasets, just pass `--testset` arg with `'bloodmnist/breastmnist/'`. This will evaluate medmnist-c and medimeta from each modality, resulting in 4 datasets evaluation.
|
| 150 |
+
|
| 151 |
+
If you need all modality datasets, you can find them as follows:
|
| 152 |
+
* [MedMNIST datasets](https://zenodo.org/records/10519652) | In-Domain _Fine-Tune_ Datasets
|
| 153 |
+
* [MediMeta datasets](https://zenodo.org/records/7884735) | OOD-B2N _Eval_ Datasets
|
| 154 |
+
* [MedMNIST-C datasets](https://github.com/francescodisalvo05/medmnistc-api) | OOD-Corruptions _Eval_ Datasets
|
| 155 |
+
|
| 156 |
## License
|
| 157 |
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
|
| 158 |
|
| 159 |
+
### Citation
|
| 160 |
+
If you find this work useful, please cite the arXiv version below:
|
| 161 |
+
```
|
| 162 |
+
@misc{imam2025t3testtimemodelmerging,
|
| 163 |
+
title={T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis},
|
| 164 |
+
author={Raza Imam and Hu Wang and Dwarikanath Mahapatra and Mohammad Yaqub},
|
| 165 |
+
year={2025},
|
| 166 |
+
eprint={2510.27265},
|
| 167 |
+
archivePrefix={arXiv},
|
| 168 |
+
primaryClass={cs.CV},
|
| 169 |
+
url={https://arxiv.org/abs/2510.27265},
|
| 170 |
+
}
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
## Contact
|
| 174 |
+
For questions or collaborations, contact [Raza Imam](mailto:[email protected]). Please feel free to raise an issue in facing error in reproducing the results.
|