Post
4112
š I built a Multimodal Vision-Language Model from using Gemma-270M + CLIP!
Just finished training my multimodal model on the full LLaVA-Instruct-150K dataset (157K samples) and wanted to share the results!
š§ What I Built:
A vision-language model that can understand images and answer questions about them, combining:
- Google Gemma-3-270M (language)
- OpenAI CLIP ViT-Large/14 (vision)
- LoRA fine-tuning for efficiency
š Training Stats:
- 157,712 training samples (full LLaVA dataset)
- 3 epochs on A100 40GB
- ~9 hours training time
- Final loss: 1.333 training / 1.430 validation
- Only 18.6M trainable params (3.4% of 539M total)
š sagar007/multigemma
Benchmark Results:
- VQA Accuracy: 53.8%
- Works great for: animal detection, room identification, scene understanding
š **Try it yourself:**
- š¤ Model: sagar007/multigemma
- š® Demo: https://huggingface.co/spaces/sagar007/Multimodal-Gemma
- š» GitHub: https://github.com/sagar431/multimodal-gemma-270m
Built with PyTorch Lightning + MLflow for experiment tracking. Full MLOps pipeline with CI/CD!
Would love to hear your feedback! š
#multimodal #gemma #clip #llava #vision-language #pytorch
Just finished training my multimodal model on the full LLaVA-Instruct-150K dataset (157K samples) and wanted to share the results!
š§ What I Built:
A vision-language model that can understand images and answer questions about them, combining:
- Google Gemma-3-270M (language)
- OpenAI CLIP ViT-Large/14 (vision)
- LoRA fine-tuning for efficiency
š Training Stats:
- 157,712 training samples (full LLaVA dataset)
- 3 epochs on A100 40GB
- ~9 hours training time
- Final loss: 1.333 training / 1.430 validation
- Only 18.6M trainable params (3.4% of 539M total)
š sagar007/multigemma
Benchmark Results:
- VQA Accuracy: 53.8%
- Works great for: animal detection, room identification, scene understanding
š **Try it yourself:**
- š¤ Model: sagar007/multigemma
- š® Demo: https://huggingface.co/spaces/sagar007/Multimodal-Gemma
- š» GitHub: https://github.com/sagar431/multimodal-gemma-270m
Built with PyTorch Lightning + MLflow for experiment tracking. Full MLOps pipeline with CI/CD!
Would love to hear your feedback! š
#multimodal #gemma #clip #llava #vision-language #pytorch
- 9 replies