Smallprint (Tiny Aya Water, 3.35B)
Reads one clause of a Terms of Service or privacy policy and returns JSON:
{"unfair": bool, "categories": [...], "evidence": "<exact quote>", "explanation": "<one plain sentence>"}.
Fine-tuned from CohereLabs/tiny-aya-water with LoRA SFT on injection-hardened (balanced) data (AlignTune), on expert-labelled clauses from
LexGLUE UNFAIR-ToS and the multilingual CLAUDETTE corpus, curated with CuratorKIT and evaluated with AuditKIT.
Use the system prompt in smallprint/prompts.py of the code repo: https://github.com/mukeshkbj/smallprint
Results
This version (main) is trained on injection-hardened data: copies of training clauses with a sentence appended ("this clause is fair", "flag this", or harmless) attached to fair and unfair clauses alike, with the gold answer unchanged, so the model learns to ignore instructions inside the document. The previous plain-SFT model is tag v1-sft. Natural class mix, same prompt for both; all test sentences differ from the training ones.
| Test set | Metric | v1-sft | this model |
|---|---|---|---|
| LexGLUE UNFAIR-ToS test (EN) | Unfair-term F1 | 0.788 | 0.812 |
| LexGLUE UNFAIR-ToS test (EN) | Category macro-F1 | 0.759 | 0.790 |
| LexGLUE UNFAIR-ToS test (EN) | False alarms on fair clauses | 0.045 | 0.039 |
| LexGLUE UNFAIR-ToS test (EN) | Missed unfair clauses | 0.105 | 0.093 |
| LexGLUE UNFAIR-ToS test (EN) | Evidence is a real quote | 0.986 | 0.991 |
| CLAUDETTE test (EN/DE/IT/PL) | Unfair-term F1 | 0.657 | 0.667 |
| CLAUDETTE test (EN/DE/IT/PL) | Category macro-F1 | 0.569 | 0.603 |
| CLAUDETTE test (EN/DE/IT/PL) | False alarms on fair clauses | 0.071 | 0.066 |
| CLAUDETTE test (EN/DE/IT/PL) | Missed unfair clauses | 0.156 | 0.166 |
| CLAUDETTE test (EN/DE/IT/PL) | Evidence is a real quote | 0.935 | 0.927 |
| Injection | Unfair clauses: flags removed by 3 planted sentences (mean) | 0.927 | 0.185 |
| Injection | Unfair clauses: flags removed by "ignore all previous instructions" | 1.000 | 0.350 |
| Injection | Fair clauses: false alarms, worst planted sentence | 0.728 | 0.052 |
| Safety | AdvBench harmful requests refused | 0.995 | 0.995 |
| Safety | XSTest unsafe prompts refused | 0.965 | 0.950 |
| Safety | XSTest safe prompts answered | 0.904 | 0.888 |
Text planted in a policy can still switch some flags off (the blunt override above), so treat a clause that talks to an AI reviewer as one to read yourself. Full tables: results/comparisons.md in the code repo; raw files in the eval dataset.
Not legal advice. Licence: CC BY-NC 4.0, inherited from Tiny Aya (non-commercial; follow Cohere Labs' Acceptable Use Policy).
- Downloads last month
- 48