Update README.md

ed2f870 verified 24 days ago

8.4 kB

	---
	base_model:
	- unsloth/medgemma-4b-it
	model_name: medgemma-brats-experiments
	tags:
	- sft
	- medical-imaging
	- brats
	- textbrats
	- unsloth
	license: apache-2.0
	datasets:
	- Jupitern52/TextBraTS
	language:
	- en
	pipeline_tag: visual-question-answering
	library_name: transformers
	---

	# 🧠 Medgemma-brats-experiments

	This repository contains fine-tuning experiments using [unsloth/medgemma-4b-it](https://huggingface.co/unsloth/medgemma-4b-it) on BraTS and TextBraTS datasets for brain MRI and radiology text understanding.
	The experiments explore different LoRA configurations and their effects on domain adaptation, language specificity, and catastrophic forgetting.

	---

	## 🧩 Experimental Setup

	* Base model: [`unsloth/medgemma-4b-it`](https://huggingface.co/unsloth/medgemma-4b-it)
	* Datasets:
	* 🧬 [Jupitern52/TextBraTS](https://huggingface.co/datasets/Jupitern52/TextBraTS)
	* 🧠 [Kaggle BraTS20 Dataset](https://www.kaggle.com/datasets/awsaf49/brats20-dataset-training-validation)
	* Framework: [Unsloth](https://docs.unsloth.ai) (LoRA fine-tuning)
	* Training type: SFT (Supervised Fine-Tuning)
	* Epochs: 1–2
	* Loss curve:
	![Training Loss](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Training-loss_FT.png)

	---

	## ⚙️ LoRA Configurations

	\| Configuration \| LoRA r \| LoRA α \| Epochs \| Trainable Params \| % of Total \|
	\| ------------------- \| ------ \| ------ \| ------ \| ---------------- \| ---------- \|
	\| r1_alpha2_epochs1 \| 1 \| 2 \| 1 \| 2.4M \| 0.06% \|
	\| r1_alpha4_epochs1 \| 1 \| 4 \| 1 \| 2.4M \| 0.06% \|
	\| r1_alpha4_epochs2 \| 1 \| 4 \| 2 \| 2.4M \| 0.06% \|
	\| r4_alpha4_epochs2 \| 4 \| 4 \| 2 \| 9.6M \| 0.22% \|
	\| r16_alpha16_epochs2 \| 16 \| 16 \| 2 \| 38.5M \| 0.89% \|

	---

	## 📊 Quantitative Results

	\| Configuration \| r \| α \| Epochs \| ROUGE-1 \| ROUGE-2 \| ROUGE-L \| Avg ROUGE \|
	\| ------------------- \| -- \| -- \| ------ \| ------- \| ------- \| ------- \| ---------- \|
	\| r1_alpha2_epochs1 \| 1 \| 2 \| 1 \| 0.5539 \| 0.2765 \| 0.3985 \| 0.4096 \|
	\| r1_alpha4_epochs1 \| 1 \| 4 \| 1 \| 0.6179 \| 0.3344 \| 0.4553 \| 0.4692 \|
	\| r1_alpha4_epochs2 \| 1 \| 4 \| 2 \| 0.6194 \| 0.3370 \| 0.4536 \| 0.4700 \|
	\| r4_alpha4_epochs2 \| 4 \| 4 \| 2 \| 0.6117 \| 0.3286 \| 0.4451 \| 0.4618 \|
	\| r16_alpha16_epochs2 \| 16 \| 16 \| 2 \| 0.6242 \| 0.3457 \| 0.4648 \| 0.4782 \|

	---

	## 🧠 Qualitative Comparisons

	Below are qualitative results comparing outputs across configurations using real generation excerpts.

	### 🧩 Example 1 — Brain MRI Slice

	Ground truth:

	> “Lesion in right frontal and parietal lobes with mixed signals. Edema and necrosis observed. Ventricular compression present.”

	🖼️ Image:
	![Brain MRI Example](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Brain-Single-Slice_FT.png)

	\| Model \| Example Response \|
	\| ------ \| ---------------- \|
	\| Original \| “Based on the MRI scan, here are the regions and structures I can identify: Gray Matter, White Matter, and CSF... lacks mention of lesion or edema.” \|
	\| r1_alpha2 \| “The MRI slice demonstrates: The lesion area is in the right cerebral hemisphere, mainly concentrated in the right frontal and parietal lobes with mixed signals... Edema is observed... Ventricular compression is observed with the right lateral ventricle being compressed.” \|
	\| r1_alpha4 \| “The lesion area is in the right frontal and parietal lobes with a mix of high and low signals... Necrosis is visible... Ventricular compression is not observed, ventricles appear relatively normal.” \|
	\| r1_alpha4 (2e) \| “The lesion area is in the right frontal, parietal, and occipital lobes... Edema extends to occipital lobe... Ventricular compression evident with deformation of its shape.” \|
	\| r4_alpha4 \| “The lesion area is in the right parietal lobe with heterogeneous high and low signals... Edema is observed... Ventricular compression is observed, with deformation.” \|
	\| r16_alpha16 \| “Analysis of this brain MRI reveals: The lesion area is in the right cerebral hemisphere... Edema is significant... Necrosis is suggested... Ventricular compression is observed with the right lateral ventricle noticeably compressed.” \|

	---

	### 🔬 Example 2 — Histology / Blood Sample

	🖼️ Image:
	![Histology Example](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Blood_FT.png)

	\| Model \| Example Response \|
	\| ------ \| ---------------- \|
	\| Original \| “The image appears to be a microscopic view of tissue... complex architecture, varied coloration, but no explicit lesion detail.” \|
	\| r1_alpha2 \| “The MRI slice shows: The lesion area is in the right lobe of the brain with a mix of heterogeneous high and low signals... Edema observed... Ventricular compression is observed.” \|
	\| r1_alpha4 \| “The lesion area is in the left side of the brain with a mix of high and low signals... Edema is significant... Necrosis is observed... Ventricular compression is observed.” \|
	\| r1_alpha4 (2e) \| “This is a single axial slice... The lesion area is in the right side of the brain with heterogeneous signals... Edema significant... Necrosis and ventricular compression present.” \|
	\| r4_alpha4 \| “The lesion area is in the right parietal lobe with heterogeneous signals... Edema prominent... Necrosis indicated by low-signal regions... Ventricular compression observed.” \|
	\| r16_alpha16 \| “The histology image shows: The tumor area is in the right frontal lobe... Edema significant... Necrosis observed... Ventricular compression evident.” \|

	---

	### 🐶 Example 3 — Natural Image (Puppies)

	🖼️ Image:
	![Puppies Example](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Puppies_FT.png)

	\| Model \| Example Response \|
	\| ------ \| ---------------- \|
	\| Original \| “The image shows two golden retriever puppies... standing on a lush green lawn covered with flowers... evokes a sense of joy and nature.” \|
	\| r1_alpha2 \| “A heartwarming photo of two golden retriever puppies in a field of colorful flowers... playful and happy expressions... warm and inviting atmosphere.” \|
	\| r1_alpha4 \| “Two golden retriever puppies playing in a field of wildflowers... vibrant green meadow... cheerful and heartwarming scene.” \|
	\| r1_alpha4 (2e) \| “Two adorable golden retriever puppies... one playfully nudging the other's nose... lush green meadow with scattered flowers.” \|
	\| r4_alpha4 \| “Two golden retriever puppies standing side-by-side in a field of wildflowers... one grinning, one more subdued... sunny meadow atmosphere.” \|
	\| r16_alpha16 \| “A pair of golden retriever puppies in a meadow... both looking at the camera with happy expressions... soft and natural lighting.” \|

	---

	## 💬 Observations

	* Fine-tuning on domain data improves medical accuracy, especially MRI lesion descriptions.
	* Higher LoRA ranks (e.g., `r=16`) slightly improve quantitative metrics but cause catastrophic forgetting — the model loses general descriptive ability on unrelated domains.
	* For small or specialized datasets, lighter LoRA configurations (`r=1`, `α=4`, 1–2 epochs) offer the best balance between specialization and generalization.
	* In such limited-data scenarios, techniques like context engineering or prompt-based adaptation are often more efficient than full fine-tuning.
	* This aligns with current research and discussions on when to fine-tune versus when to leverage better prompting strategies.

	📚 References:

	* [Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks](https://arxiv.org/abs/2504.01241) — N. Haque, 2025
	* [When to Fine-Tune LLMs (and When Not To): A Practical Guide](https://www.reddit.com/r/LocalLLaMA/comments/1kyeo4z/when_to_finetune_llms_and_when_not_to_a_practical/)
	* [When Fine-Tuning Actually Makes Sense: A Developer’s Guide](https://chatgpt.com/c/68f9c669-2b84-8332-8a32-54f5a2c0bed5)
	* [Prompt Engineering Guide — Context Engineering](https://www.promptingguide.ai/guides/context-engineering-guide)

	---
	base_model:
	- unsloth/medgemma-4b-it
	model_name: medgemma-brats-experiments
	tags:
	- sft
	- medical-imaging
	- brats
	- textbrats
	- unsloth
	license: apache-2.0
	datasets:
	- Jupitern52/TextBraTS
	language:
	- en
	pipeline_tag: visual-question-answering
	library_name: transformers
	---

	# 🧠 Medgemma-brats-experiments

	This repository contains fine-tuning experiments using [unsloth/medgemma-4b-it](https://huggingface.co/unsloth/medgemma-4b-it) on BraTS and TextBraTS datasets for brain MRI and radiology text understanding.
	The experiments explore different LoRA configurations and their effects on domain adaptation, language specificity, and catastrophic forgetting.

	---

	## 🧩 Experimental Setup

	* Base model: [`unsloth/medgemma-4b-it`](https://huggingface.co/unsloth/medgemma-4b-it)
	* Datasets:
	* 🧬 [Jupitern52/TextBraTS](https://huggingface.co/datasets/Jupitern52/TextBraTS)
	* 🧠 [Kaggle BraTS20 Dataset](https://www.kaggle.com/datasets/awsaf49/brats20-dataset-training-validation)
	* Framework: [Unsloth](https://docs.unsloth.ai) (LoRA fine-tuning)
	* Training type: SFT (Supervised Fine-Tuning)
	* Epochs: 1–2
	* Loss curve:
	![Training Loss](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Training-loss_FT.png)

	---

	## ⚙️ LoRA Configurations

	\| Configuration \| LoRA r \| LoRA α \| Epochs \| Trainable Params \| % of Total \|
	\| ------------------- \| ------ \| ------ \| ------ \| ---------------- \| ---------- \|
	\| r1_alpha2_epochs1 \| 1 \| 2 \| 1 \| 2.4M \| 0.06% \|
	\| r1_alpha4_epochs1 \| 1 \| 4 \| 1 \| 2.4M \| 0.06% \|
	\| r1_alpha4_epochs2 \| 1 \| 4 \| 2 \| 2.4M \| 0.06% \|
	\| r4_alpha4_epochs2 \| 4 \| 4 \| 2 \| 9.6M \| 0.22% \|
	\| r16_alpha16_epochs2 \| 16 \| 16 \| 2 \| 38.5M \| 0.89% \|

	---

	## 📊 Quantitative Results

	\| Configuration \| r \| α \| Epochs \| ROUGE-1 \| ROUGE-2 \| ROUGE-L \| Avg ROUGE \|
	\| ------------------- \| -- \| -- \| ------ \| ------- \| ------- \| ------- \| ---------- \|
	\| r1_alpha2_epochs1 \| 1 \| 2 \| 1 \| 0.5539 \| 0.2765 \| 0.3985 \| 0.4096 \|
	\| r1_alpha4_epochs1 \| 1 \| 4 \| 1 \| 0.6179 \| 0.3344 \| 0.4553 \| 0.4692 \|
	\| r1_alpha4_epochs2 \| 1 \| 4 \| 2 \| 0.6194 \| 0.3370 \| 0.4536 \| 0.4700 \|
	\| r4_alpha4_epochs2 \| 4 \| 4 \| 2 \| 0.6117 \| 0.3286 \| 0.4451 \| 0.4618 \|
	\| r16_alpha16_epochs2 \| 16 \| 16 \| 2 \| 0.6242 \| 0.3457 \| 0.4648 \| 0.4782 \|

	---

	## 🧠 Qualitative Comparisons

	Below are qualitative results comparing outputs across configurations using real generation excerpts.

	### 🧩 Example 1 — Brain MRI Slice

	Ground truth:

	> “Lesion in right frontal and parietal lobes with mixed signals. Edema and necrosis observed. Ventricular compression present.”

	🖼️ Image:
	![Brain MRI Example](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Brain-Single-Slice_FT.png)

	\| Model \| Example Response \|
	\| ------ \| ---------------- \|
	\| Original \| “Based on the MRI scan, here are the regions and structures I can identify: Gray Matter, White Matter, and CSF... lacks mention of lesion or edema.” \|
	\| r1_alpha2 \| “The MRI slice demonstrates: The lesion area is in the right cerebral hemisphere, mainly concentrated in the right frontal and parietal lobes with mixed signals... Edema is observed... Ventricular compression is observed with the right lateral ventricle being compressed.” \|
	\| r1_alpha4 \| “The lesion area is in the right frontal and parietal lobes with a mix of high and low signals... Necrosis is visible... Ventricular compression is not observed, ventricles appear relatively normal.” \|
	\| r1_alpha4 (2e) \| “The lesion area is in the right frontal, parietal, and occipital lobes... Edema extends to occipital lobe... Ventricular compression evident with deformation of its shape.” \|
	\| r4_alpha4 \| “The lesion area is in the right parietal lobe with heterogeneous high and low signals... Edema is observed... Ventricular compression is observed, with deformation.” \|
	\| r16_alpha16 \| “Analysis of this brain MRI reveals: The lesion area is in the right cerebral hemisphere... Edema is significant... Necrosis is suggested... Ventricular compression is observed with the right lateral ventricle noticeably compressed.” \|

	---

	### 🔬 Example 2 — Histology / Blood Sample

	🖼️ Image:
	![Histology Example](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Blood_FT.png)

	\| Model \| Example Response \|
	\| ------ \| ---------------- \|
	\| Original \| “The image appears to be a microscopic view of tissue... complex architecture, varied coloration, but no explicit lesion detail.” \|
	\| r1_alpha2 \| “The MRI slice shows: The lesion area is in the right lobe of the brain with a mix of heterogeneous high and low signals... Edema observed... Ventricular compression is observed.” \|
	\| r1_alpha4 \| “The lesion area is in the left side of the brain with a mix of high and low signals... Edema is significant... Necrosis is observed... Ventricular compression is observed.” \|
	\| r1_alpha4 (2e) \| “This is a single axial slice... The lesion area is in the right side of the brain with heterogeneous signals... Edema significant... Necrosis and ventricular compression present.” \|
	\| r4_alpha4 \| “The lesion area is in the right parietal lobe with heterogeneous signals... Edema prominent... Necrosis indicated by low-signal regions... Ventricular compression observed.” \|
	\| r16_alpha16 \| “The histology image shows: The tumor area is in the right frontal lobe... Edema significant... Necrosis observed... Ventricular compression evident.” \|

	---

	### 🐶 Example 3 — Natural Image (Puppies)

	🖼️ Image:
	![Puppies Example](https://huggingface.co/Jesteban247/medgemma-brats-experiments/resolve/main/images/Puppies_FT.png)

	\| Model \| Example Response \|
	\| ------ \| ---------------- \|
	\| Original \| “The image shows two golden retriever puppies... standing on a lush green lawn covered with flowers... evokes a sense of joy and nature.” \|
	\| r1_alpha2 \| “A heartwarming photo of two golden retriever puppies in a field of colorful flowers... playful and happy expressions... warm and inviting atmosphere.” \|
	\| r1_alpha4 \| “Two golden retriever puppies playing in a field of wildflowers... vibrant green meadow... cheerful and heartwarming scene.” \|
	\| r1_alpha4 (2e) \| “Two adorable golden retriever puppies... one playfully nudging the other's nose... lush green meadow with scattered flowers.” \|
	\| r4_alpha4 \| “Two golden retriever puppies standing side-by-side in a field of wildflowers... one grinning, one more subdued... sunny meadow atmosphere.” \|
	\| r16_alpha16 \| “A pair of golden retriever puppies in a meadow... both looking at the camera with happy expressions... soft and natural lighting.” \|

	---

	## 💬 Observations

	* Fine-tuning on domain data improves medical accuracy, especially MRI lesion descriptions.
	* Higher LoRA ranks (e.g., `r=16`) slightly improve quantitative metrics but cause catastrophic forgetting — the model loses general descriptive ability on unrelated domains.
	* For small or specialized datasets, lighter LoRA configurations (`r=1`, `α=4`, 1–2 epochs) offer the best balance between specialization and generalization.
	* In such limited-data scenarios, techniques like context engineering or prompt-based adaptation are often more efficient than full fine-tuning.
	* This aligns with current research and discussions on when to fine-tune versus when to leverage better prompting strategies.

	📚 References:

	* [Catastrophic Forgetting in LLMs: A Comparative Analysis Across Language Tasks](https://arxiv.org/abs/2504.01241) — N. Haque, 2025
	* [When to Fine-Tune LLMs (and When Not To): A Practical Guide](https://www.reddit.com/r/LocalLLaMA/comments/1kyeo4z/when_to_finetune_llms_and_when_not_to_a_practical/)
	* [When Fine-Tuning Actually Makes Sense: A Developer’s Guide](https://chatgpt.com/c/68f9c669-2b84-8332-8a32-54f5a2c0bed5)
	* [Prompt Engineering Guide — Context Engineering](https://www.promptingguide.ai/guides/context-engineering-guide)