Initial: 5-language Indic experts + Hinglish router + all eval JSONs

Browse files

Files changed (21) hide show

README.md +206 -0
bn_best_bn_v2.pt +3 -0
en_router_best_v5.pt +3 -0
eval_bn_pure_fleurs.json +0 -0
eval_hi_fleurs_samples.json +0 -0
eval_hi_pure_fleurs.json +0 -0
eval_mr_pure_fleurs.json +0 -0
eval_ta_pure_fleurs.json +0 -0
eval_te_pure_fleurs.json +0 -0
eval_vanilla_fleurs_bn.json +0 -0
eval_vanilla_fleurs_hi.json +0 -0
eval_vanilla_fleurs_mr.json +0 -0
eval_vanilla_fleurs_ta.json +0 -0
eval_vanilla_fleurs_te.json +0 -0
fleurs_normalized_results.json +132 -0
hi_best_v5.pt +3 -0
hi_router_best_v5.pt +3 -0
mr_best_mr.pt +3 -0
router_best_v5.pt +3 -0
ta_best_ta.pt +3 -0
te_best_te_v2.pt +3 -0

README.md ADDED Viewed

	@@ -0,0 +1,206 @@

+---
+language:
+- en
+- hi
+- ta
+- te
+- bn
+- mr
+license: mit
+library_name: pytorch
+pipeline_tag: automatic-speech-recognition
+tags:
+- whisper
+- lora
+- mixture-of-experts
+- asr
+- speech-recognition
+- indic
+- tamil
+- telugu
+- bengali
+- marathi
+- hindi
+- hinglish
+- code-switching
+- low-resource
+- edge-asr
+- devanagari
+metrics:
+- wer
+- cer
+- fuzzy-wer
+model-index:
+- name: polywhisper-hinglish-router
+  results:
+  - task:
+      type: automatic-speech-recognition
+      name: Speech Recognition
+    dataset:
+      name: PolyWhisper Hinglish Test (3,129 utterances)
+      type: eulogik/polywhisper-hinglish
+      split: test
+    metrics:
+    - type: wer
+      value: 58.8
+      name: WER
+    - type: cer
+      value: 57.9
+      name: CER
+- name: polywhisper-indic-experts
+  results:
+  - task:
+      type: automatic-speech-recognition
+      name: Speech Recognition
+    dataset:
+      name: FLEURS (hi/ta/te/bn/mr)
+      type: google/fleurs
+    metrics:
+    - type: wer
+      value: 38.5
+      name: WER (hi)
+    - type: cer
+      value: 14.4
+      name: CER (hi)
+    - type: wer
+      value: 73.9
+      name: WER (ta)
+    - type: cer
+      value: 25.6
+      name: CER (ta)
+    - type: wer
+      value: 82.7
+      name: WER (te)
+    - type: cer
+      value: 32.7
+      name: CER (te)
+    - type: wer
+      value: 84.9
+      name: WER (bn)
+    - type: cer
+      value: 54.3
+      name: CER (bn)
+    - type: wer
+      value: 65.0
+      name: WER (mr)
+    - type: cer
+      value: 22.9
+      name: CER (mr)
+---
+# PolyWhisper: Per-Language LoRA Experts on Shared Whisper-Base
+**5 languages · 6 experts · 1 frozen 74M encoder · 0 encoder retraining**
+PolyWhisper is a family of tiny language-specific LoRA experts on a frozen Whisper-Base encoder. It tackles a problem vanilla Whisper-Base cannot even begin to solve: **producing correct-script text for Indian languages**.
+## The script-confusion finding
+Vanilla Whisper-Base doesn't just perform worse on Indic languages — it emits the **wrong script entirely** (Urdu-Arabic for Telugu/Bengali/Marathi, 0–0.5% correct-script rate). The same encoder produces near-perfect Devanagari for Hindi but collapses to Arabic for the other Indic scripts.
+Per-language LoRA experts on the shared frozen encoder fix this completely.
+## Architecture
+```
+Audio (16kHz) → Whisper-Base Encoder [frozen, 74M]
+                         ↓
+          Per-language LoRA adapter [rank-16, ~0.8M each]
+                         ↓
+          Transcription (correct-script, correct language)
+```
+- **Backbone**: openai/whisper-base — frozen encoder + frozen base decoder
+- **Experts**: rank-16 LoRA on decoder cross-attention K/V projections
+- **Router (Hinglish only)**: 33K-param MLP on decoder hidden states, per-token en/hi selection
+## Results
+### Hinglish code-switch (v5 router, 3,129 utterances)
+| System | WER | FuzzyWER | CER | Hallucinations |
+|---|---|---|---|---|
+| **PolyWhisper v5** | **58.8%** | **57.3%** | **57.9%** | **13** |
+| Vanilla Whisper-Base | 66.6% | 63.3% | 67.5% | 279 |
+| Static 50/50 mix | 72.1% | 70.4% | 71.1% | 655 |
+### Indic experts (FLEURS, ortho-normalized, script-matched, 256 decode tokens)
+| Lang | Vanilla script-match | Expert script-match | Pure expert WER/CER |
+|---|---|---|---|
+| **Hindi** | 6.2% | **99.8%** | **38.5 / 14.4** |
+| **Tamil** | 96.4% | **99.3%** | 73.9 / 25.6 |
+| **Telugu** | 0.0% | **92.6%** | 82.7 / 32.7 |
+| **Bengali** | 0.0% | **77.1%** | 84.9 / 54.3 |
+| **Marathi** | 0.5% | **99.8%** | 65.0 / 22.9 |
+Vanilla scores for te/bn/mr are unreportable (0–0.5% correct-script output). Experts restore correct-script and cut CER 10–22 points. Residual WER is the 39M-parameter ceiling.
+## Files
+### Hinglish router
+| File | What |
+|---|---|
+| `en_router_best_v5.pt` | English LoRA expert (rank-8 decoder) |
+| `hi_router_best_v5.pt` | Hindi LoRA expert with encoder LoRA |
+| `router_best_v5.pt` | Per-token router (33K params) |
+| `hi_best_v5.pt` | Hindi pure expert (encoder + decoder LoRA) |
+### Indic experts
+| File | What |
+|---|---|
+| `ta_best_ta.pt` | Tamil expert (rank-16 decoder) |
+| `te_best_te_v2.pt` | Telugu expert (rank-16 decoder) |
+| `bn_best_bn_v2.pt` | Bengali expert (rank-16 decoder) |
+| `mr_best_mr.pt` | Marathi expert (rank-16 decoder) |
+### Evaluation
+| File | What |
+|---|---|
+| `fleurs_normalized_results.json` | Ortho-normalized scores for all 5 languages |
+| `eval_*_fleurs.json` | Per-sample results (rescorable) |
+| `eval_*_samples.json` | Per-sample Hinglish results |
+## Usage (Indic expert example)
+```python
+import torch
+from transformers import WhisperProcessor, WhisperForConditionalGeneration
+from peft import PeftModel
+proc = WhisperProcessor.from_pretrained("openai/whisper-base")
+model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
+model = PeftModel.from_pretrained(model, "eulogik/polywhisper", "ta_best_ta.pt")
+model.eval()
+audio = ...  # 16kHz mono
+feats = proc.feature_extractor([audio], sampling_rate=16000, return_tensors="pt").input_features
+out = model.generate(feats, max_new_tokens=256, language="ta", task="transcribe")
+print(proc.decode(out[0], skip_special_tokens=True))
+```
+## Training
+- 3 epochs, LR 1e-4, rank-16 LoRA on decoder cross-attention K/V
+- Data: IndicVoices-ST (~19–20K samples, gated) + FLEURS (test)
+- Trained on M4 Mac Mini 16GB (Apple MPS, fp16)
+- Training time: ~3h per language, fully automated (`train_v3.py --langs ta`)
+## Key finding for the paper
+> Language-specific LoRA experts on a shared frozen encoder fix Whisper-Base's script-confusion problem for 5 Indian languages. Vanilla outputs Urdu-Arabic for Telugu/Bengali/Marathi (0% correct-script); experts restore 77–100% correct-script output. Hindi expert reaches 38.5% WER / 14.4% CER — near-usable for edge deployment.
+## License
+MIT. © 2026 [Eulogik](https://huggingface.co/eulogik). Whisper is OpenAI's model. Data from IndicVoices-ST (CC-BY) and FLEURS (CC-BY 4.0).
+## Citation
+```bibtex
+@misc{eulogik2026polywhisper,
+  title={PolyWhisper: Per-Language LoRA Experts on Shared Encoder for Indic ASR},
+  author={Eulogik},
+  year={2026},
+  howpublished={\url{https://huggingface.co/eulogik/polywhisper}},
+  note={MIT licensed; 5 Indian languages + Hinglish router; script-confusion fix}
+}
+```

bn_best_bn_v2.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:94f32fed12f63ca9230edbd63729f1332e43dd2f1626a217a884508132c232b8
+size 3178791

en_router_best_v5.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e88c782489a10da76e84720d2fd94c708089df3d5ec0fbffd8ee7f27d55f5730
+size 3179071

eval_bn_pure_fleurs.json ADDED Viewed