|
|
--- |
|
|
language: |
|
|
- en |
|
|
license: apache-2.0 |
|
|
base_model: google/flan-t5-base |
|
|
tags: |
|
|
- text2text-generation |
|
|
- summarization |
|
|
- xsum |
|
|
- lora |
|
|
- peft |
|
|
datasets: |
|
|
- EdinburghNLP/xsum |
|
|
metrics: |
|
|
- rouge |
|
|
--- |
|
|
|
|
|
# FLAN-T5-Base Fine-tuned on XSum with LoRA |
|
|
|
|
|
This model is a fine-tuned version of [google/flan-t5-base](https://huggingface.co/google/flan-t5-base) on the [XSum dataset](https://huggingface.co/datasets/EdinburghNLP/xsum) using **LoRA (Low-Rank Adaptation)** for parameter-efficient fine-tuning. |
|
|
|
|
|
## Model Description |
|
|
|
|
|
- **Base Model:** google/flan-t5-base |
|
|
- **Task:** Extreme Summarization (one-sentence summaries) |
|
|
- **Dataset:** XSum (BBC news articles) |
|
|
- **Training Method:** LoRA (Low-Rank Adaptation) |
|
|
- **Parameters:** 0.00M trainable (0.00% of 249.35M total) |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### LoRA Configuration |
|
|
- **Rank (r):** 16 |
|
|
- **Alpha:** 32 |
|
|
- **Target modules:** q, v |
|
|
- **Dropout:** 0.05 |
|
|
|
|
|
### Training Hyperparameters |
|
|
- **Learning rate:** 3e-4 |
|
|
- **Batch size:** 8 |
|
|
- **Epochs:** 3 |
|
|
- **Optimizer:** AdamW |
|
|
- **Mixed precision:** FP16 |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
|
|
from peft import PeftModel |
|
|
|
|
|
# Load base model and tokenizer |
|
|
base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base") |
|
|
tokenizer = AutoTokenizer.from_pretrained("Ekansh112/flan-t5-base-xsum-lora") |
|
|
|
|
|
# Load LoRA adapters |
|
|
model = PeftModel.from_pretrained(base_model, "Ekansh112/flan-t5-base-xsum-lora") |
|
|
|
|
|
# Generate summary |
|
|
text = "Your article text here..." |
|
|
inputs = tokenizer("summarize: " + text, return_tensors="pt", max_length=512, truncation=True) |
|
|
outputs = model.generate(**inputs, max_length=64, num_beams=4, length_penalty=2.0) |
|
|
summary = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
print(summary) |
|
|
``` |
|
|
|
|
|
## Performance |
|
|
|
|
|
Evaluation metrics on XSum test set: |
|
|
- **ROUGE-1:** [Add your score] |
|
|
- **ROUGE-2:** [Add your score] |
|
|
- **ROUGE-L:** [Add your score] |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model, please cite the original FLAN-T5 paper and the XSum dataset: |
|
|
|
|
|
```bibtex |
|
|
@article{chung2022scaling, |
|
|
title={Scaling instruction-finetuned language models}, |
|
|
author={Chung, Hyung Won and others}, |
|
|
journal={arXiv preprint arXiv:2210.11416}, |
|
|
year={2022} |
|
|
} |
|
|
|
|
|
@inproceedings{narayan2018don, |
|
|
title={Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization}, |
|
|
author={Narayan, Shashi and others}, |
|
|
booktitle={EMNLP}, |
|
|
year={2018} |
|
|
} |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
This model inherits the license from the base model: Apache 2.0 |
|
|
|
|
|
--- |
|
|
|
|
|
**Trained by:** Ekansh112 |
|
|
|
