Create README.md

README.md

@@ -0,0 +1,63 @@
+---
+language: 
+  - en
+tags:
+- retrieval
+- math-retrieval
+datasets:
+- MathematicalStackExchange
+- ARQMath
+---
+
+# ALBERT for ARQMath 3
+
+This repository contains our best model for ARQMath 3, the math_10 model. It was initialised from ALBERT-base-v2 and further pre-trained on Math StackExchange in three different stages. We also added more LaTeX tokens to the tokenizer to enable a better tokenization of mathematical formulas. math_10 was fine-tuned on a classification task to determine whether a given question (sequence 1) matches a given answer (sequence 2). The classification output can be used for ranking the best answers. For further details, please read our paper: http://ceur-ws.org/Vol-3180/paper-07.pdf. We plan on also publishing the other fine-tuned models as well as the base models. Links to these repositories will be added here soon.
+
+# Usage
+
+```python
+# based on https://huggingface.co/docs/transformers/main/en/task_summary#sequence-classification
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+tokenizer = AutoTokenizer.from_pretrained("AnReu/albert-for-arqmath-3")
+
+model = AutoModelForSequenceClassification.from_pretrained("AnReu/albert-for-arqmath-3")
+
+classes = ["non relevant", "relevant"]
+
+sequence_0 = "How can I calculate x in $3x = 5$"
+sequence_1 = "Just divide by 3: $x = \\frac{5}{3}$"
+sequence_2 = "The general rule for squaring a sum is $(a+b)^2=a^2+2ab+b^2$"
+
+# The tokenizer will automatically add any model specific separators (i.e. <CLS> and <SEP>) and tokens to
+# the sequence, as well as compute the attention masks.
+irrelevant = tokenizer(sequence_0, sequence_2, return_tensors="pt")
+relevant = tokenizer(sequence_0, sequence_1, return_tensors="pt")
+
+irrelevant_classification_logits = model(**irrelevant).logits
+relevant_classification_logits = model(**relevant).logits
+
+irrelevant_results = torch.softmax(irrelevant_classification_logits, dim=1).tolist()[0]
+relevant_results = torch.softmax(relevant_classification_logits, dim=1).tolist()[0]
+
+# Should be irrelevant
+for i in range(len(classes)):
+    print(f"{classes[i]}: {int(round(irrelevant_results[i] * 100))}%")
+
+# Should be relevant
+for i in range(len(classes)):
+    print(f"{classes[i]}: {int(round(relevant_results[i] * 100))}%")
+
+```
+
+# Citation
+
+If you find this model useful, consider citing our paper:
+```
+@article{reusch2022transformer,
+  title={Transformer-Encoder and Decoder Models for Questions on Math},
+  author={Reusch, Anja and Thiele, Maik and Lehner, Wolfgang},
+  year={2022},
+  organization={CLEF}
+}
+```