Update README.md

README.md

@@ -32,28 +32,44 @@ More information needed
 
 More information needed
 
-## Training and evaluation data
-
-More information needed
 
 ## Training procedure
 
 ### Training hyperparameters
 
-**TODO**: update the values below
-
-The following hyperparameters were used during training:
-- learning_rate: 5e-05
-- train_batch_size: 8
-- eval_batch_size: 8
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- num_epochs: 3.0
+This model was trained with the code available on the [parameterlab/apricot GitHub repository](https://github.com/parameterlab/apricot) using the following command:
+```shell
+python3 run_regression_experiment.py --model-identifier lmsys/vicuna-7b-v1.5 --dataset-name coqa --device cuda:0 --num-training-steps 600 --num-in-context-samples 0 --data-dir $data_dir --model-save-dir $model_save_dir --result-dir $result_dir --lr 0.00008836 --weight-decay 0.0007421 --push-to-hub
+```
 
 ### Framework versions
 
 - Transformers 4.32.0
 - Pytorch 2.0.0+cu117
 - Datasets 2.14.6
-- Tokenizers 0.13.3
+- Tokenizers 0.13.3
+
+## Citation
+If you find 🍑 Apricot models useful for your work, please cite our paper:
+``` latex
+@inproceedings{ulmer-etal-2024-calibrating,
+    title = "Calibrating Large Language Models Using Their Generations Only",
+    author = "Ulmer, Dennis  and
+      Gubri, Martin  and
+      Lee, Hwaran  and
+      Yun, Sangdoo  and
+      Oh, Seong",
+    editor = "Ku, Lun-Wei  and
+      Martins, Andre  and
+      Srikumar, Vivek",
+    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
+    month = aug,
+    year = "2024",
+    address = "Bangkok, Thailand",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2024.acl-long.824",
+    doi = "10.18653/v1/2024.acl-long.824",
+    pages = "15440--15459",
+    abstract = "As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model{'}s confidence in its prediction becomes even more important. However, finding effective ways to calibrate LLMs{---}especially when the only interface to the models is their generated text{---}remains a challenge. We propose APRICOT (Auxiliary prediction of confidence targets): A method to set confidence targets and train an additional model that predicts an LLM{'}s confidence based on its textual input and output alone. This approach has several advantages: It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages, for instance by verbalizing the predicted confidence or using it to re-prompting the LLM to accurately reflecting its uncertainty. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.",
+}
+```