pysentimiento/bertweet-irony

Irony detection in English

bertweet-irony

Repository: https://github.com/pysentimiento/pysentimiento/

Model trained with SemEval 2018 dataset Task 3 (Van Hee et all, 2018) for irony detection. Base model is [BERTweet], a RoBERTa model trained in English tweets.

The positive class marks irony, the negative class marks not ironic content.

Results

Results for the four tasks evaluated in pysentimiento. Results are expressed as Macro F1 scores

Model	sentiment	emotion	hate_speech	irony
bert	69.6 +- 0.4	42.7 +- 0.6	56.0 +- 0.8	68.1 +- 2.2
electra	70.9 +- 0.4	37.2 +- 2.9	55.6 +- 0.6	71.3 +- 1.8
roberta	70.4 +- 0.3	45.0 +- 0.9	55.1 +- 0.4	70.4 +- 2.9
robertuito	69.6 +- 0.5	43.0 +- 3.3	57.5 +- 0.2	73.9 +- 1.4
bertweet	72.0 +- 0.4	43.1 +- 1.8	57.7 +- 0.7	80.8 +- 0.7

Note that for Hate Speech, these are the results for Semeval 2019, Task 5 Subtask B (HS+TR+AG detection)

Citation

If you use this model in your research, please cite pysentimiento, dataset and pre-trained model papers:

@misc{perez2021pysentimiento,
      title={pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks},
      author={Juan Manuel Pérez and Juan Carlos Giudici and Franco Luque},
      year={2021},
      eprint={2106.09462},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
@inproceedings{van2018semeval,
  title={Semeval-2018 task 3: Irony detection in english tweets},
  author={Van Hee, Cynthia and Lefever, Els and Hoste, V{\'e}ronique},
  booktitle={Proceedings of The 12th International Workshop on Semantic Evaluation},
  pages={39--50},
  year={2018}
}

@inproceedings{nguyen2020bertweet,
  title={BERTweet: A pre-trained language model for English Tweets},
  author={Nguyen, Dat Quoc and Vu, Thanh and Nguyen, Anh Tuan},
  booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  pages={9--14},
  year={2020}
}