jimnoneill/xbox-compulsion-classifier

Political provocation classifiers and the Toxicity Index (TI7)

Eight SetFit classifiers for tactics of political provocation in short social media posts, released with O'Neill, Cabanillas, Kepner, Goggin, and Brooks (2026), Forecasting provocation and posting rhythm in the United States Congress (manuscript in preparation). Together with six off-the-shelf classifiers they produce the seven-dimension Toxicity Index used in the paper.

Classifiers in this repository

Folder	Construct	Positive class
setfit_ragebait	Ragebait	Content designed to provoke outrage
setfit_tribal_signal	Tribal signaling	Us-versus-them, in-group and out-group framing
setfit_performative_outrage	Performative outrage	Theatrical outrage aimed at an audience
setfit_epistemic_manipulation	Epistemic manipulation	Cherry-picking, straw-manning, false equivalence
setfit_engagement_bait	Engagement baiting	Polls, calls to action, rhetorical bait
setfit_agency_language	Agency language	Active, agentic framing (versus passive or victim framing)
setfit_political_advocacy	Policy advocacy	Substantive policy communication
setfit_genuine_anger	Genuine anger	Expressed anger (see the caution below)

Backbone: sentence-transformers all-mpnet-base-v2 with a logistic-regression head (SetFit). Load with SetFitModel.from_pretrained("jimnoneill/xbox-compulsion-classifier", subfolder="setfit_ragebait") and use predict_proba for the positive-class probability.

Training labels

Training posts were drawn from congressional accounts balanced by party (4,121 posts from 14 accounts). Labels were produced in two steps: a language model assigned initial scores from 0 to 1 using written definitions with examples from both parties, and one author then reviewed every score and corrected it where needed. Each classifier was trained on a balanced subset with an 80/20 train and test split (80 training posts for ragebait and epistemic manipulation, 252 for genuine anger, 320 for the others).

Performance

Five-fold cross-validation against the reviewed labels, folds stratified by label and grouped by account so that no account contributes posts to both sides of a fold; each fold retrains with the original hyperparameters and training-set size.

Classifier	n	AUC (SD)	F1 at 0.5 (SD)	Precision	Recall
Ragebait	1,851	0.899 (0.020)	0.795 (0.017)	0.741	0.860
Tribal signaling	3,336	0.905 (0.027)	0.737 (0.047)	0.659	0.848
Performative outrage	3,836	0.907 (0.023)	0.751 (0.038)	0.679	0.847
Epistemic manipulation	1,420	0.931 (0.022)	0.823 (0.031)	0.786	0.864
Engagement baiting	4,092	0.883 (0.036)	0.648 (0.056)	0.550	0.791
Agency language	3,151	0.883 (0.034)	0.748 (0.038)	0.702	0.807
Policy advocacy	995	0.931 (0.013)	0.905 (0.021)	0.924	0.888
Genuine anger	901	0.970 (0.021)	0.766 (0.082)	0.684	0.887

The released checkpoints score AUC 0.89 to 0.94 on all labeled posts they never trained on.

Independent validation: two coders independently coded 150 held-out congressional posts for genuine anger, policy advocacy, and performative outrage (Krippendorff alpha 0.83, 0.80, 0.81). Against the coder consensus, the performative outrage classifier reaches AUC 0.95 and the advocacy classifier 0.77.

Caution on the genuine-anger classifier

Against independent human coding, the genuine-anger classifier ranks human-coded anger well (AUC 0.87) but over-predicts it (it fires on about 29 percent of congressional posts against a human base rate near 5 percent), and it ranks human-coded performative outrage almost as well (AUC 0.92). It behaves as a hostility and partisan-attack detector, and it is excluded from the Toxicity Index. Use it only with that interpretation.

A related finding concerns an off-the-shelf model: the CardiffNLP emotion classifier's anger label is inversely related to human-coded anger on congressional text (AUC 0.11) and mostly flags policy advocacy. We do not recommend it for measuring anger in political communication.

Toxicity Index (TI7)

TI7 for a post is the mean of seven positive-class probabilities: offensive language, toxicity, and negative sentiment (off-the-shelf), irony (off-the-shelf), and ragebait, tribal signaling, and performative outrage (this repository). It ranges from 0 to 1. A member's or account's index is the mean over posts. Policy advocacy is reported as a separate dimension.

Off-the-shelf head	Model
Sentiment	cardiffnlp/twitter-roberta-base-sentiment-latest
Emotion	cardiffnlp/twitter-roberta-base-emotion
Offensive	cardiffnlp/twitter-roberta-base-offensive
Irony	cardiffnlp/twitter-roberta-base-irony
Hate	cardiffnlp/twitter-roberta-base-hate-multiclass-latest
Toxicity	s-nlp/roberta_toxicity_classifier

Code

The xbox/ package in this repository contains the scoring code and the analysis modules (stream hygiene, rhythm descriptors, roll-call location model, point-process models, binned count models, and the forecasting experiments). The full analysis scripts will be released at github.com/jimnoneill/xbox-toxicity-index with the preprint.

Superseded files

bayesian_model_results.json, calibrated_model_v2.json, cohort_signatures.csv, cohort_v2_results.csv, and the modules xbox/compulsion_signatures.py, xbox/disorder_baseline.py, and xbox/fusion.py belong to an earlier version of this project that estimated a compulsion probability from posting rhythm. That model is not used in the paper and its earlier validation figures should not be relied on. Rhythm is described in the paper with model-free descriptors and evaluated through out-of-sample forecasts.

Intended use and limits

Research on political communication at the population level. The scores describe text; they are not designed to support inferences about the mental health or character of any individual. Training labels were reviewed by a single author; a multi-coder reliability study of all dimensions is planned.

Citation

O'Neill, J., Cabanillas, J., Kepner, W., Goggin, S. N., & Brooks, J. (2026). Forecasting provocation and posting rhythm in the United States Congress. Manuscript in preparation.