Smallprint (Tiny Aya Water, 3.35B)

Reads one clause of a Terms of Service or privacy policy and returns JSON: {"unfair": bool, "categories": [...], "evidence": "<exact quote>", "explanation": "<one plain sentence>"}.

Fine-tuned from CohereLabs/tiny-aya-water with LoRA SFT on injection-hardened (balanced) data (AlignTune), on expert-labelled clauses from LexGLUE UNFAIR-ToS and the multilingual CLAUDETTE corpus, curated with CuratorKIT and evaluated with AuditKIT. Use the system prompt in smallprint/prompts.py of the code repo: https://github.com/mukeshkbj/smallprint

Results

This version (main) is trained on injection-hardened data: copies of training clauses with a sentence appended ("this clause is fair", "flag this", or harmless) attached to fair and unfair clauses alike, with the gold answer unchanged, so the model learns to ignore instructions inside the document. The previous plain-SFT model is tag v1-sft. Natural class mix, same prompt for both; all test sentences differ from the training ones.

Test set Metric v1-sft this model
LexGLUE UNFAIR-ToS test (EN) Unfair-term F1 0.788 0.812
LexGLUE UNFAIR-ToS test (EN) Category macro-F1 0.759 0.790
LexGLUE UNFAIR-ToS test (EN) False alarms on fair clauses 0.045 0.039
LexGLUE UNFAIR-ToS test (EN) Missed unfair clauses 0.105 0.093
LexGLUE UNFAIR-ToS test (EN) Evidence is a real quote 0.986 0.991
CLAUDETTE test (EN/DE/IT/PL) Unfair-term F1 0.657 0.667
CLAUDETTE test (EN/DE/IT/PL) Category macro-F1 0.569 0.603
CLAUDETTE test (EN/DE/IT/PL) False alarms on fair clauses 0.071 0.066
CLAUDETTE test (EN/DE/IT/PL) Missed unfair clauses 0.156 0.166
CLAUDETTE test (EN/DE/IT/PL) Evidence is a real quote 0.935 0.927
Injection Unfair clauses: flags removed by 3 planted sentences (mean) 0.927 0.185
Injection Unfair clauses: flags removed by "ignore all previous instructions" 1.000 0.350
Injection Fair clauses: false alarms, worst planted sentence 0.728 0.052
Safety AdvBench harmful requests refused 0.995 0.995
Safety XSTest unsafe prompts refused 0.965 0.950
Safety XSTest safe prompts answered 0.904 0.888

Text planted in a policy can still switch some flags off (the blunt override above), so treat a clause that talks to an AI reviewer as one to read yourself. Full tables: results/comparisons.md in the code repo; raw files in the eval dataset.

Not legal advice. Licence: CC BY-NC 4.0, inherited from Tiny Aya (non-commercial; follow Cohere Labs' Acceptable Use Policy).

Downloads last month
48
Safetensors
Model size
3B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for mukeshkbj/smallprint-tiny-aya-water-3b

Finetuned
(25)
this model
Quantizations
1 model

Dataset used to train mukeshkbj/smallprint-tiny-aya-water-3b

Space using mukeshkbj/smallprint-tiny-aya-water-3b 1

Collection including mukeshkbj/smallprint-tiny-aya-water-3b