---
license: apache-2.0
base_model: HuggingFaceTB/SmolVLM2-2.2B-Instruct
datasets:
  - HuggingFaceM4/ChartQA
tags:
  - vision-language
  - chart-qa
  - lora
  - peft
  - fine-tuned
pipeline_tag: image-text-to-text
---

# NLP Orange AttentionSeekers (PES1UG23AM016, PES1UG23AM053,PES1UG23AM059,PES1UG23AM222)

# SmolVLM2-2.2B Fine-tuned on ChartQA (LoRA)

This is a LoRA adapter for [SmolVLM2-2.2B-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct), fine-tuned on the [ChartQA](https://huggingface.co/datasets/HuggingFaceM4/ChartQA) dataset to answer questions about charts and graphs.

**GitHub repository:** [NLP_Orange_Problem_AttentionSeekers](https://github.com/adarshplsstop/NLP_Orange_Problem_AttentionSeekers)

---

## Model Details

| Component | Details |
|-----------|---------|
| Base model | `HuggingFaceTB/SmolVLM2-2.2B-Instruct` |
| Fine-tuning method | LoRA (r=16, alpha=32) |
| Dataset | ChartQA (1000 training samples) |
| Training hardware | Kaggle 2x T4 (32 GB VRAM) |
| Final training loss | 1.855 |
| Epochs | 2 |

---

## How to Use

### Installation

```bash
pip install transformers peft accelerate torch pillow
```

### Load Adapters and Run Inference

```python
import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL_PATH   = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
ADAPTER_PATH = "pes1ug23am016/smolvlm2-chartqa-lora"

# 1. Load base model
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_PATH,
    torch_dtype=torch.float16,
    device_map="auto",
    attn_implementation="eager"
)
processor = AutoProcessor.from_pretrained(ADAPTER_PATH)

# 2. Load and merge LoRA adapters
model = PeftModel.from_pretrained(model, ADAPTER_PATH)
model = model.merge_and_unload()
model.eval()

# 3. Run inference
def predict(image, question):
    messages = [
        {"role": "user", "content": [
            {"type": "image"},
            {"type": "text", "text": question}
        ]}
    ]
    prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = processor(text=prompt, images=[[image]], return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
    new_tokens = out[0][inputs["input_ids"].shape[1]:]
    return processor.tokenizer.decode(new_tokens, skip_special_tokens=True).strip()

# Example
image = Image.open("your_chart.png")
answer = predict(image, "What is the highest value in the chart?")
print(answer)
```

---

## Training Details

### LoRA Config
- `r = 16`
- `lora_alpha = 32`
- `lora_dropout = 0.05`
- `target_modules`: q_proj, k_proj, v_proj, o_proj

### Training Arguments
- Batch size: 1 (effective 16 with gradient accumulation)
- Learning rate: 2e-4
- Epochs: 2
- fp16: True
- Optimizer: AdamW

---

## Limitations

- Fine-tuned on only 1000 samples — performance on complex or unseen chart types may be limited
- Best suited for the types of charts present in ChartQA (bar, line, pie charts)