Hugging Face's logo

Sean13
/
mistral-7b-instruct-v0.2-cpo-full

Text Generation
Transformers
TensorBoard
Safetensors
mistral
Generated from Trainer
trl
cpo
conversational
text-generation-inference
Model card Files
xet
Metrics Community
Sean13 commited on
Commit
ea8c0b6
·
verified ·
1 Parent(s): e3856e9

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +9 -0
  3. generation_config.json +6 -0
  4. train_results.json +9 -0
  5. trainer_state.json +846 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: mistralai/Mistral-7B-Instruct-v0.2
3
+ library_name: transformers
4
+ model_name: mistral-7b-instruct-v0.2-cpo-full
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - cpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for mistral-7b-instruct-v0.2-cpo-full
13
+
14
+ This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="Sean13/mistral-7b-instruct-v0.2-cpo-full", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+
31
+
32
+ This model was trained with CPO, a method introduced in [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://huggingface.co/papers/2401.08417).
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.12.2
37
+ - Transformers: 4.46.3
38
+ - Pytorch: 2.7.1
39
+ - Datasets: 4.0.0
40
+ - Tokenizers: 0.20.3
41
+
42
+ ## Citations
43
+
44
+ Cite CPO as:
45
+
46
+ ```bibtex
47
+ @inproceedings{xu2024contrastive,
48
+ title = {{Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation}},
49
+ author = {Haoran Xu and Amr Sharaf and Yunmo Chen and Weiting Tan and Lingfeng Shen and Benjamin Van Durme and Kenton Murray and Young Jin Kim},
50
+ year = 2024,
51
+ booktitle = {Forty-first International Conference on Machine Learning, {ICML} 2024, Vienna, Austria, July 21-27, 2024},
52
+ publisher = {OpenReview.net},
53
+ url = {https://openreview.net/forum?id=51iwkioZpn}
54
+ }
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 1.0373174434555674,
5
+ "train_runtime": 5792.1601,
6
+ "train_samples": 59720,
7
+ "train_samples_per_second": 10.31,
8
+ "train_steps_per_second": 0.081
9
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "transformers_version": "4.46.3"
6
+ }
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 1.0,
3
+ "total_flos": 0.0,
4
+ "train_loss": 1.0373174434555674,
5
+ "train_runtime": 5792.1601,
6
+ "train_samples": 59720,
7
+ "train_samples_per_second": 10.31,
8
+ "train_steps_per_second": 0.081
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,846 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 1.0,
5
+ "eval_steps": 100,
6
+ "global_step": 467,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.021413276231263382,
13
+ "grad_norm": 28.65919274184648,
14
+ "learning_rate": 1.0638297872340425e-07,
15
+ "logits/chosen": -2.390625,
16
+ "logits/rejected": -2.390625,
17
+ "logps/chosen": -202.0,
18
+ "logps/rejected": -207.0,
19
+ "loss": 1.1772,
20
+ "nll_loss": 0.482421875,
21
+ "rewards/accuracies": 0.5375000238418579,
22
+ "rewards/chosen": -2.015625,
23
+ "rewards/margins": 0.06005859375,
24
+ "rewards/rejected": -2.078125,
25
+ "step": 10
26
+ },
27
+ {
28
+ "epoch": 0.042826552462526764,
29
+ "grad_norm": 27.82635851268732,
30
+ "learning_rate": 2.127659574468085e-07,
31
+ "logits/chosen": -2.4375,
32
+ "logits/rejected": -2.421875,
33
+ "logps/chosen": -163.0,
34
+ "logps/rejected": -172.0,
35
+ "loss": 1.1776,
36
+ "nll_loss": 0.4453125,
37
+ "rewards/accuracies": 0.5687500238418579,
38
+ "rewards/chosen": -1.6328125,
39
+ "rewards/margins": 0.08251953125,
40
+ "rewards/rejected": -1.71875,
41
+ "step": 20
42
+ },
43
+ {
44
+ "epoch": 0.06423982869379015,
45
+ "grad_norm": 25.319570642507898,
46
+ "learning_rate": 3.1914893617021275e-07,
47
+ "logits/chosen": -2.375,
48
+ "logits/rejected": -2.359375,
49
+ "logps/chosen": -179.0,
50
+ "logps/rejected": -177.0,
51
+ "loss": 1.1491,
52
+ "nll_loss": 0.439453125,
53
+ "rewards/accuracies": 0.4749999940395355,
54
+ "rewards/chosen": -1.7890625,
55
+ "rewards/margins": -0.015869140625,
56
+ "rewards/rejected": -1.7734375,
57
+ "step": 30
58
+ },
59
+ {
60
+ "epoch": 0.08565310492505353,
61
+ "grad_norm": 29.77277195802064,
62
+ "learning_rate": 4.25531914893617e-07,
63
+ "logits/chosen": -2.171875,
64
+ "logits/rejected": -2.15625,
65
+ "logps/chosen": -148.0,
66
+ "logps/rejected": -160.0,
67
+ "loss": 1.114,
68
+ "nll_loss": 0.380859375,
69
+ "rewards/accuracies": 0.550000011920929,
70
+ "rewards/chosen": -1.4765625,
71
+ "rewards/margins": 0.119140625,
72
+ "rewards/rejected": -1.6015625,
73
+ "step": 40
74
+ },
75
+ {
76
+ "epoch": 0.10706638115631692,
77
+ "grad_norm": 31.775152726182768,
78
+ "learning_rate": 4.999370587356267e-07,
79
+ "logits/chosen": -2.109375,
80
+ "logits/rejected": -2.140625,
81
+ "logps/chosen": -159.0,
82
+ "logps/rejected": -171.0,
83
+ "loss": 1.0965,
84
+ "nll_loss": 0.42578125,
85
+ "rewards/accuracies": 0.5687500238418579,
86
+ "rewards/chosen": -1.59375,
87
+ "rewards/margins": 0.1103515625,
88
+ "rewards/rejected": -1.703125,
89
+ "step": 50
90
+ },
91
+ {
92
+ "epoch": 0.1284796573875803,
93
+ "grad_norm": 28.626004581134318,
94
+ "learning_rate": 4.988189843662815e-07,
95
+ "logits/chosen": -2.21875,
96
+ "logits/rejected": -2.203125,
97
+ "logps/chosen": -142.0,
98
+ "logps/rejected": -153.0,
99
+ "loss": 1.098,
100
+ "nll_loss": 0.376953125,
101
+ "rewards/accuracies": 0.574999988079071,
102
+ "rewards/chosen": -1.4140625,
103
+ "rewards/margins": 0.109375,
104
+ "rewards/rejected": -1.5234375,
105
+ "step": 60
106
+ },
107
+ {
108
+ "epoch": 0.14989293361884368,
109
+ "grad_norm": 26.61794395631547,
110
+ "learning_rate": 4.963094133060148e-07,
111
+ "logits/chosen": -2.25,
112
+ "logits/rejected": -2.25,
113
+ "logps/chosen": -148.0,
114
+ "logps/rejected": -147.0,
115
+ "loss": 1.0737,
116
+ "nll_loss": 0.3515625,
117
+ "rewards/accuracies": 0.4937500059604645,
118
+ "rewards/chosen": -1.484375,
119
+ "rewards/margins": -0.0174560546875,
120
+ "rewards/rejected": -1.46875,
121
+ "step": 70
122
+ },
123
+ {
124
+ "epoch": 0.17130620985010706,
125
+ "grad_norm": 29.279588800218576,
126
+ "learning_rate": 4.924223800941717e-07,
127
+ "logits/chosen": -2.203125,
128
+ "logits/rejected": -2.1875,
129
+ "logps/chosen": -135.0,
130
+ "logps/rejected": -142.0,
131
+ "loss": 1.0499,
132
+ "nll_loss": 0.3671875,
133
+ "rewards/accuracies": 0.5375000238418579,
134
+ "rewards/chosen": -1.3515625,
135
+ "rewards/margins": 0.06982421875,
136
+ "rewards/rejected": -1.421875,
137
+ "step": 80
138
+ },
139
+ {
140
+ "epoch": 0.19271948608137046,
141
+ "grad_norm": 30.174466010531273,
142
+ "learning_rate": 4.871796225971999e-07,
143
+ "logits/chosen": -2.328125,
144
+ "logits/rejected": -2.3125,
145
+ "logps/chosen": -143.0,
146
+ "logps/rejected": -148.0,
147
+ "loss": 1.072,
148
+ "nll_loss": 0.3828125,
149
+ "rewards/accuracies": 0.543749988079071,
150
+ "rewards/chosen": -1.4296875,
151
+ "rewards/margins": 0.049072265625,
152
+ "rewards/rejected": -1.4765625,
153
+ "step": 90
154
+ },
155
+ {
156
+ "epoch": 0.21413276231263384,
157
+ "grad_norm": 26.53002947944615,
158
+ "learning_rate": 4.806104604416823e-07,
159
+ "logits/chosen": -2.3125,
160
+ "logits/rejected": -2.3125,
161
+ "logps/chosen": -131.0,
162
+ "logps/rejected": -144.0,
163
+ "loss": 1.0527,
164
+ "nll_loss": 0.328125,
165
+ "rewards/accuracies": 0.574999988079071,
166
+ "rewards/chosen": -1.3125,
167
+ "rewards/margins": 0.1279296875,
168
+ "rewards/rejected": -1.4375,
169
+ "step": 100
170
+ },
171
+ {
172
+ "epoch": 0.21413276231263384,
173
+ "eval_logits/chosen": -2.359375,
174
+ "eval_logits/rejected": -2.359375,
175
+ "eval_logps/chosen": -138.0,
176
+ "eval_logps/rejected": -140.0,
177
+ "eval_loss": 1.0688878297805786,
178
+ "eval_nll_loss": 0.373046875,
179
+ "eval_rewards/accuracies": 0.5159574747085571,
180
+ "eval_rewards/chosen": -1.3828125,
181
+ "eval_rewards/margins": 0.0201416015625,
182
+ "eval_rewards/rejected": -1.40625,
183
+ "eval_runtime": 65.4847,
184
+ "eval_samples_per_second": 45.721,
185
+ "eval_steps_per_second": 0.718,
186
+ "step": 100
187
+ },
188
+ {
189
+ "epoch": 0.23554603854389722,
190
+ "grad_norm": 29.144988092850184,
191
+ "learning_rate": 4.7275163104709194e-07,
192
+ "logits/chosen": -2.28125,
193
+ "logits/rejected": -2.265625,
194
+ "logps/chosen": -125.5,
195
+ "logps/rejected": -139.0,
196
+ "loss": 1.0484,
197
+ "nll_loss": 0.33203125,
198
+ "rewards/accuracies": 0.606249988079071,
199
+ "rewards/chosen": -1.2578125,
200
+ "rewards/margins": 0.1396484375,
201
+ "rewards/rejected": -1.390625,
202
+ "step": 110
203
+ },
204
+ {
205
+ "epoch": 0.2569593147751606,
206
+ "grad_norm": 26.63402391939837,
207
+ "learning_rate": 4.636470841752404e-07,
208
+ "logits/chosen": -2.3125,
209
+ "logits/rejected": -2.296875,
210
+ "logps/chosen": -143.0,
211
+ "logps/rejected": -148.0,
212
+ "loss": 1.0552,
213
+ "nll_loss": 0.34375,
214
+ "rewards/accuracies": 0.5625,
215
+ "rewards/chosen": -1.4296875,
216
+ "rewards/margins": 0.05419921875,
217
+ "rewards/rejected": -1.484375,
218
+ "step": 120
219
+ },
220
+ {
221
+ "epoch": 0.278372591006424,
222
+ "grad_norm": 28.998527842665165,
223
+ "learning_rate": 4.533477361453819e-07,
224
+ "logits/chosen": -2.375,
225
+ "logits/rejected": -2.390625,
226
+ "logps/chosen": -130.0,
227
+ "logps/rejected": -134.0,
228
+ "loss": 1.0539,
229
+ "nll_loss": 0.359375,
230
+ "rewards/accuracies": 0.5375000238418579,
231
+ "rewards/chosen": -1.296875,
232
+ "rewards/margins": 0.046875,
233
+ "rewards/rejected": -1.34375,
234
+ "step": 130
235
+ },
236
+ {
237
+ "epoch": 0.29978586723768735,
238
+ "grad_norm": 27.402556855452524,
239
+ "learning_rate": 4.4191118508950277e-07,
240
+ "logits/chosen": -2.3125,
241
+ "logits/rejected": -2.3125,
242
+ "logps/chosen": -143.0,
243
+ "logps/rejected": -143.0,
244
+ "loss": 1.0161,
245
+ "nll_loss": 0.33984375,
246
+ "rewards/accuracies": 0.48124998807907104,
247
+ "rewards/chosen": -1.4375,
248
+ "rewards/margins": -0.002166748046875,
249
+ "rewards/rejected": -1.4296875,
250
+ "step": 140
251
+ },
252
+ {
253
+ "epoch": 0.32119914346895073,
254
+ "grad_norm": 30.364999796943756,
255
+ "learning_rate": 4.294013888402029e-07,
256
+ "logits/chosen": -2.390625,
257
+ "logits/rejected": -2.359375,
258
+ "logps/chosen": -134.0,
259
+ "logps/rejected": -155.0,
260
+ "loss": 1.0322,
261
+ "nll_loss": 0.345703125,
262
+ "rewards/accuracies": 0.6000000238418579,
263
+ "rewards/chosen": -1.34375,
264
+ "rewards/margins": 0.208984375,
265
+ "rewards/rejected": -1.5546875,
266
+ "step": 150
267
+ },
268
+ {
269
+ "epoch": 0.3426124197002141,
270
+ "grad_norm": 31.21988192093287,
271
+ "learning_rate": 4.158883072525528e-07,
272
+ "logits/chosen": -2.359375,
273
+ "logits/rejected": -2.359375,
274
+ "logps/chosen": -139.0,
275
+ "logps/rejected": -145.0,
276
+ "loss": 1.0367,
277
+ "nll_loss": 0.33984375,
278
+ "rewards/accuracies": 0.5249999761581421,
279
+ "rewards/chosen": -1.390625,
280
+ "rewards/margins": 0.0625,
281
+ "rewards/rejected": -1.453125,
282
+ "step": 160
283
+ },
284
+ {
285
+ "epoch": 0.3640256959314775,
286
+ "grad_norm": 28.50764905524101,
287
+ "learning_rate": 4.0144751096020497e-07,
288
+ "logits/chosen": -2.40625,
289
+ "logits/rejected": -2.390625,
290
+ "logps/chosen": -128.0,
291
+ "logps/rejected": -141.0,
292
+ "loss": 1.0343,
293
+ "nll_loss": 0.322265625,
294
+ "rewards/accuracies": 0.5874999761581421,
295
+ "rewards/chosen": -1.28125,
296
+ "rewards/margins": 0.1279296875,
297
+ "rewards/rejected": -1.40625,
298
+ "step": 170
299
+ },
300
+ {
301
+ "epoch": 0.3854389721627409,
302
+ "grad_norm": 26.699112025919906,
303
+ "learning_rate": 3.8615975875375676e-07,
304
+ "logits/chosen": -2.390625,
305
+ "logits/rejected": -2.390625,
306
+ "logps/chosen": -134.0,
307
+ "logps/rejected": -145.0,
308
+ "loss": 1.0066,
309
+ "nll_loss": 0.33203125,
310
+ "rewards/accuracies": 0.5562499761581421,
311
+ "rewards/chosen": -1.34375,
312
+ "rewards/margins": 0.1015625,
313
+ "rewards/rejected": -1.4453125,
314
+ "step": 180
315
+ },
316
+ {
317
+ "epoch": 0.4068522483940043,
318
+ "grad_norm": 30.132705246813664,
319
+ "learning_rate": 3.7011054594483443e-07,
320
+ "logits/chosen": -2.4375,
321
+ "logits/rejected": -2.421875,
322
+ "logps/chosen": -132.0,
323
+ "logps/rejected": -139.0,
324
+ "loss": 1.0171,
325
+ "nll_loss": 0.32421875,
326
+ "rewards/accuracies": 0.518750011920929,
327
+ "rewards/chosen": -1.328125,
328
+ "rewards/margins": 0.058837890625,
329
+ "rewards/rejected": -1.3828125,
330
+ "step": 190
331
+ },
332
+ {
333
+ "epoch": 0.4282655246252677,
334
+ "grad_norm": 28.668330159886178,
335
+ "learning_rate": 3.5338962624163016e-07,
336
+ "logits/chosen": -2.390625,
337
+ "logits/rejected": -2.40625,
338
+ "logps/chosen": -143.0,
339
+ "logps/rejected": -151.0,
340
+ "loss": 1.0257,
341
+ "nll_loss": 0.3359375,
342
+ "rewards/accuracies": 0.5375000238418579,
343
+ "rewards/chosen": -1.421875,
344
+ "rewards/margins": 0.08056640625,
345
+ "rewards/rejected": -1.5078125,
346
+ "step": 200
347
+ },
348
+ {
349
+ "epoch": 0.4282655246252677,
350
+ "eval_logits/chosen": -2.421875,
351
+ "eval_logits/rejected": -2.421875,
352
+ "eval_logps/chosen": -135.0,
353
+ "eval_logps/rejected": -137.0,
354
+ "eval_loss": 1.057234287261963,
355
+ "eval_nll_loss": 0.36328125,
356
+ "eval_rewards/accuracies": 0.5265957713127136,
357
+ "eval_rewards/chosen": -1.34375,
358
+ "eval_rewards/margins": 0.027099609375,
359
+ "eval_rewards/rejected": -1.375,
360
+ "eval_runtime": 65.5097,
361
+ "eval_samples_per_second": 45.703,
362
+ "eval_steps_per_second": 0.717,
363
+ "step": 200
364
+ },
365
+ {
366
+ "epoch": 0.44967880085653106,
367
+ "grad_norm": 28.48793224061069,
368
+ "learning_rate": 3.360905098097587e-07,
369
+ "logits/chosen": -2.40625,
370
+ "logits/rejected": -2.390625,
371
+ "logps/chosen": -115.0,
372
+ "logps/rejected": -126.0,
373
+ "loss": 1.0308,
374
+ "nll_loss": 0.330078125,
375
+ "rewards/accuracies": 0.574999988079071,
376
+ "rewards/chosen": -1.1484375,
377
+ "rewards/margins": 0.11328125,
378
+ "rewards/rejected": -1.265625,
379
+ "step": 210
380
+ },
381
+ {
382
+ "epoch": 0.47109207708779444,
383
+ "grad_norm": 29.54686623645261,
384
+ "learning_rate": 3.1830994032548e-07,
385
+ "logits/chosen": -2.390625,
386
+ "logits/rejected": -2.40625,
387
+ "logps/chosen": -133.0,
388
+ "logps/rejected": -137.0,
389
+ "loss": 1.0299,
390
+ "nll_loss": 0.357421875,
391
+ "rewards/accuracies": 0.550000011920929,
392
+ "rewards/chosen": -1.328125,
393
+ "rewards/margins": 0.03955078125,
394
+ "rewards/rejected": -1.375,
395
+ "step": 220
396
+ },
397
+ {
398
+ "epoch": 0.4925053533190578,
399
+ "grad_norm": 28.690918035629924,
400
+ "learning_rate": 3.001473539458182e-07,
401
+ "logits/chosen": -2.359375,
402
+ "logits/rejected": -2.328125,
403
+ "logps/chosen": -141.0,
404
+ "logps/rejected": -147.0,
405
+ "loss": 1.0372,
406
+ "nll_loss": 0.36328125,
407
+ "rewards/accuracies": 0.550000011920929,
408
+ "rewards/chosen": -1.4140625,
409
+ "rewards/margins": 0.06298828125,
410
+ "rewards/rejected": -1.46875,
411
+ "step": 230
412
+ },
413
+ {
414
+ "epoch": 0.5139186295503212,
415
+ "grad_norm": 30.99889420788846,
416
+ "learning_rate": 2.817043232212371e-07,
417
+ "logits/chosen": -2.453125,
418
+ "logits/rejected": -2.453125,
419
+ "logps/chosen": -123.0,
420
+ "logps/rejected": -140.0,
421
+ "loss": 1.0132,
422
+ "nll_loss": 0.3203125,
423
+ "rewards/accuracies": 0.6187499761581421,
424
+ "rewards/chosen": -1.234375,
425
+ "rewards/margins": 0.171875,
426
+ "rewards/rejected": -1.40625,
427
+ "step": 240
428
+ },
429
+ {
430
+ "epoch": 0.5353319057815846,
431
+ "grad_norm": 29.718698452984967,
432
+ "learning_rate": 2.63083989060736e-07,
433
+ "logits/chosen": -2.4375,
434
+ "logits/rejected": -2.4375,
435
+ "logps/chosen": -141.0,
436
+ "logps/rejected": -148.0,
437
+ "loss": 1.0288,
438
+ "nll_loss": 0.3515625,
439
+ "rewards/accuracies": 0.606249988079071,
440
+ "rewards/chosen": -1.40625,
441
+ "rewards/margins": 0.0791015625,
442
+ "rewards/rejected": -1.484375,
443
+ "step": 250
444
+ },
445
+ {
446
+ "epoch": 0.556745182012848,
447
+ "grad_norm": 28.738200181119513,
448
+ "learning_rate": 2.4439048392604877e-07,
449
+ "logits/chosen": -2.40625,
450
+ "logits/rejected": -2.40625,
451
+ "logps/chosen": -124.0,
452
+ "logps/rejected": -135.0,
453
+ "loss": 1.0068,
454
+ "nll_loss": 0.318359375,
455
+ "rewards/accuracies": 0.574999988079071,
456
+ "rewards/chosen": -1.2421875,
457
+ "rewards/margins": 0.10595703125,
458
+ "rewards/rejected": -1.3515625,
459
+ "step": 260
460
+ },
461
+ {
462
+ "epoch": 0.5781584582441114,
463
+ "grad_norm": 30.07846487699893,
464
+ "learning_rate": 2.2572834948067795e-07,
465
+ "logits/chosen": -2.34375,
466
+ "logits/rejected": -2.34375,
467
+ "logps/chosen": -126.0,
468
+ "logps/rejected": -141.0,
469
+ "loss": 1.0279,
470
+ "nll_loss": 0.32421875,
471
+ "rewards/accuracies": 0.606249988079071,
472
+ "rewards/chosen": -1.2578125,
473
+ "rewards/margins": 0.1484375,
474
+ "rewards/rejected": -1.40625,
475
+ "step": 270
476
+ },
477
+ {
478
+ "epoch": 0.5995717344753747,
479
+ "grad_norm": 30.03488169227238,
480
+ "learning_rate": 2.072019519505062e-07,
481
+ "logits/chosen": -2.40625,
482
+ "logits/rejected": -2.390625,
483
+ "logps/chosen": -127.5,
484
+ "logps/rejected": -147.0,
485
+ "loss": 1.0196,
486
+ "nll_loss": 0.34375,
487
+ "rewards/accuracies": 0.581250011920929,
488
+ "rewards/chosen": -1.2734375,
489
+ "rewards/margins": 0.1884765625,
490
+ "rewards/rejected": -1.46875,
491
+ "step": 280
492
+ },
493
+ {
494
+ "epoch": 0.6209850107066381,
495
+ "grad_norm": 28.07914335597087,
496
+ "learning_rate": 1.8891489846552644e-07,
497
+ "logits/chosen": -2.375,
498
+ "logits/rejected": -2.375,
499
+ "logps/chosen": -127.5,
500
+ "logps/rejected": -147.0,
501
+ "loss": 0.999,
502
+ "nll_loss": 0.341796875,
503
+ "rewards/accuracies": 0.5874999761581421,
504
+ "rewards/chosen": -1.2734375,
505
+ "rewards/margins": 0.1904296875,
506
+ "rewards/rejected": -1.46875,
507
+ "step": 290
508
+ },
509
+ {
510
+ "epoch": 0.6423982869379015,
511
+ "grad_norm": 26.394334611818714,
512
+ "learning_rate": 1.7096945764674398e-07,
513
+ "logits/chosen": -2.328125,
514
+ "logits/rejected": -2.3125,
515
+ "logps/chosen": -132.0,
516
+ "logps/rejected": -138.0,
517
+ "loss": 1.0239,
518
+ "nll_loss": 0.345703125,
519
+ "rewards/accuracies": 0.550000011920929,
520
+ "rewards/chosen": -1.3203125,
521
+ "rewards/margins": 0.0634765625,
522
+ "rewards/rejected": -1.3828125,
523
+ "step": 300
524
+ },
525
+ {
526
+ "epoch": 0.6423982869379015,
527
+ "eval_logits/chosen": -2.4375,
528
+ "eval_logits/rejected": -2.4375,
529
+ "eval_logps/chosen": -133.0,
530
+ "eval_logps/rejected": -136.0,
531
+ "eval_loss": 1.0522451400756836,
532
+ "eval_nll_loss": 0.357421875,
533
+ "eval_rewards/accuracies": 0.5345744490623474,
534
+ "eval_rewards/chosen": -1.328125,
535
+ "eval_rewards/margins": 0.0303955078125,
536
+ "eval_rewards/rejected": -1.359375,
537
+ "eval_runtime": 65.445,
538
+ "eval_samples_per_second": 45.748,
539
+ "eval_steps_per_second": 0.718,
540
+ "step": 300
541
+ },
542
+ {
543
+ "epoch": 0.6638115631691649,
544
+ "grad_norm": 29.46313387643739,
545
+ "learning_rate": 1.5346598767856345e-07,
546
+ "logits/chosen": -2.390625,
547
+ "logits/rejected": -2.359375,
548
+ "logps/chosen": -127.0,
549
+ "logps/rejected": -149.0,
550
+ "loss": 0.9753,
551
+ "nll_loss": 0.34375,
552
+ "rewards/accuracies": 0.625,
553
+ "rewards/chosen": -1.2734375,
554
+ "rewards/margins": 0.22265625,
555
+ "rewards/rejected": -1.4921875,
556
+ "step": 310
557
+ },
558
+ {
559
+ "epoch": 0.6852248394004282,
560
+ "grad_norm": 26.6293918608003,
561
+ "learning_rate": 1.365023750651133e-07,
562
+ "logits/chosen": -2.34375,
563
+ "logits/rejected": -2.34375,
564
+ "logps/chosen": -135.0,
565
+ "logps/rejected": -145.0,
566
+ "loss": 1.0029,
567
+ "nll_loss": 0.35546875,
568
+ "rewards/accuracies": 0.6187499761581421,
569
+ "rewards/chosen": -1.3515625,
570
+ "rewards/margins": 0.10205078125,
571
+ "rewards/rejected": -1.453125,
572
+ "step": 320
573
+ },
574
+ {
575
+ "epoch": 0.7066381156316917,
576
+ "grad_norm": 27.055527797905288,
577
+ "learning_rate": 1.201734872092077e-07,
578
+ "logits/chosen": -2.4375,
579
+ "logits/rejected": -2.40625,
580
+ "logps/chosen": -108.0,
581
+ "logps/rejected": -124.5,
582
+ "loss": 1.0136,
583
+ "nll_loss": 0.30078125,
584
+ "rewards/accuracies": 0.59375,
585
+ "rewards/chosen": -1.078125,
586
+ "rewards/margins": 0.16796875,
587
+ "rewards/rejected": -1.2421875,
588
+ "step": 330
589
+ },
590
+ {
591
+ "epoch": 0.728051391862955,
592
+ "grad_norm": 28.596848946440872,
593
+ "learning_rate": 1.0457064187534861e-07,
594
+ "logits/chosen": -2.40625,
595
+ "logits/rejected": -2.40625,
596
+ "logps/chosen": -128.0,
597
+ "logps/rejected": -147.0,
598
+ "loss": 1.0013,
599
+ "nll_loss": 0.349609375,
600
+ "rewards/accuracies": 0.612500011920929,
601
+ "rewards/chosen": -1.28125,
602
+ "rewards/margins": 0.185546875,
603
+ "rewards/rejected": -1.46875,
604
+ "step": 340
605
+ },
606
+ {
607
+ "epoch": 0.7494646680942184,
608
+ "grad_norm": 30.677711170624534,
609
+ "learning_rate": 8.978109650374396e-08,
610
+ "logits/chosen": -2.484375,
611
+ "logits/rejected": -2.453125,
612
+ "logps/chosen": -142.0,
613
+ "logps/rejected": -155.0,
614
+ "loss": 1.0174,
615
+ "nll_loss": 0.36328125,
616
+ "rewards/accuracies": 0.5687500238418579,
617
+ "rewards/chosen": -1.4140625,
618
+ "rewards/margins": 0.1318359375,
619
+ "rewards/rejected": -1.546875,
620
+ "step": 350
621
+ },
622
+ {
623
+ "epoch": 0.7708779443254818,
624
+ "grad_norm": 28.79231928911763,
625
+ "learning_rate": 7.588756023130833e-08,
626
+ "logits/chosen": -2.453125,
627
+ "logits/rejected": -2.4375,
628
+ "logps/chosen": -138.0,
629
+ "logps/rejected": -153.0,
630
+ "loss": 1.0146,
631
+ "nll_loss": 0.337890625,
632
+ "rewards/accuracies": 0.581250011920929,
633
+ "rewards/chosen": -1.3828125,
634
+ "rewards/margins": 0.1533203125,
635
+ "rewards/rejected": -1.53125,
636
+ "step": 360
637
+ },
638
+ {
639
+ "epoch": 0.7922912205567452,
640
+ "grad_norm": 28.24522355678149,
641
+ "learning_rate": 6.296773134861824e-08,
642
+ "logits/chosen": -2.421875,
643
+ "logits/rejected": -2.40625,
644
+ "logps/chosen": -124.0,
645
+ "logps/rejected": -139.0,
646
+ "loss": 1.0022,
647
+ "nll_loss": 0.314453125,
648
+ "rewards/accuracies": 0.574999988079071,
649
+ "rewards/chosen": -1.2421875,
650
+ "rewards/margins": 0.14453125,
651
+ "rewards/rejected": -1.3828125,
652
+ "step": 370
653
+ },
654
+ {
655
+ "epoch": 0.8137044967880086,
656
+ "grad_norm": 29.407093626605853,
657
+ "learning_rate": 5.109386277955477e-08,
658
+ "logits/chosen": -2.4375,
659
+ "logits/rejected": -2.421875,
660
+ "logps/chosen": -135.0,
661
+ "logps/rejected": -153.0,
662
+ "loss": 1.0115,
663
+ "nll_loss": 0.33984375,
664
+ "rewards/accuracies": 0.612500011920929,
665
+ "rewards/chosen": -1.3515625,
666
+ "rewards/margins": 0.181640625,
667
+ "rewards/rejected": -1.5390625,
668
+ "step": 380
669
+ },
670
+ {
671
+ "epoch": 0.8351177730192719,
672
+ "grad_norm": 29.01803889736997,
673
+ "learning_rate": 4.0332358013644015e-08,
674
+ "logits/chosen": -2.46875,
675
+ "logits/rejected": -2.46875,
676
+ "logps/chosen": -119.0,
677
+ "logps/rejected": -134.0,
678
+ "loss": 1.0093,
679
+ "nll_loss": 0.30859375,
680
+ "rewards/accuracies": 0.5625,
681
+ "rewards/chosen": -1.1875,
682
+ "rewards/margins": 0.146484375,
683
+ "rewards/rejected": -1.3359375,
684
+ "step": 390
685
+ },
686
+ {
687
+ "epoch": 0.8565310492505354,
688
+ "grad_norm": 28.058309152739813,
689
+ "learning_rate": 3.074339975080836e-08,
690
+ "logits/chosen": -2.453125,
691
+ "logits/rejected": -2.4375,
692
+ "logps/chosen": -115.5,
693
+ "logps/rejected": -128.0,
694
+ "loss": 1.0016,
695
+ "nll_loss": 0.306640625,
696
+ "rewards/accuracies": 0.6499999761581421,
697
+ "rewards/chosen": -1.15625,
698
+ "rewards/margins": 0.1220703125,
699
+ "rewards/rejected": -1.28125,
700
+ "step": 400
701
+ },
702
+ {
703
+ "epoch": 0.8565310492505354,
704
+ "eval_logits/chosen": -2.453125,
705
+ "eval_logits/rejected": -2.453125,
706
+ "eval_logps/chosen": -132.0,
707
+ "eval_logps/rejected": -135.0,
708
+ "eval_loss": 1.0496279001235962,
709
+ "eval_nll_loss": 0.35546875,
710
+ "eval_rewards/accuracies": 0.539893627166748,
711
+ "eval_rewards/chosen": -1.3203125,
712
+ "eval_rewards/margins": 0.0311279296875,
713
+ "eval_rewards/rejected": -1.3515625,
714
+ "eval_runtime": 65.4097,
715
+ "eval_samples_per_second": 45.773,
716
+ "eval_steps_per_second": 0.719,
717
+ "step": 400
718
+ },
719
+ {
720
+ "epoch": 0.8779443254817987,
721
+ "grad_norm": 27.582942094559826,
722
+ "learning_rate": 2.2380613335296033e-08,
723
+ "logits/chosen": -2.40625,
724
+ "logits/rejected": -2.40625,
725
+ "logps/chosen": -124.0,
726
+ "logps/rejected": -138.0,
727
+ "loss": 1.0123,
728
+ "nll_loss": 0.349609375,
729
+ "rewards/accuracies": 0.6499999761581421,
730
+ "rewards/chosen": -1.234375,
731
+ "rewards/margins": 0.1416015625,
732
+ "rewards/rejected": -1.3828125,
733
+ "step": 410
734
+ },
735
+ {
736
+ "epoch": 0.8993576017130621,
737
+ "grad_norm": 29.183346377804614,
738
+ "learning_rate": 1.5290766861003475e-08,
739
+ "logits/chosen": -2.34375,
740
+ "logits/rejected": -2.328125,
741
+ "logps/chosen": -119.0,
742
+ "logps/rejected": -135.0,
743
+ "loss": 1.0187,
744
+ "nll_loss": 0.34375,
745
+ "rewards/accuracies": 0.668749988079071,
746
+ "rewards/chosen": -1.1875,
747
+ "rewards/margins": 0.158203125,
748
+ "rewards/rejected": -1.3515625,
749
+ "step": 420
750
+ },
751
+ {
752
+ "epoch": 0.9207708779443254,
753
+ "grad_norm": 28.433546149053406,
754
+ "learning_rate": 9.513509625323518e-09,
755
+ "logits/chosen": -2.4375,
756
+ "logits/rejected": -2.4375,
757
+ "logps/chosen": -132.0,
758
+ "logps/rejected": -136.0,
759
+ "loss": 0.9987,
760
+ "nll_loss": 0.357421875,
761
+ "rewards/accuracies": 0.5062500238418579,
762
+ "rewards/chosen": -1.3203125,
763
+ "rewards/margins": 0.045166015625,
764
+ "rewards/rejected": -1.359375,
765
+ "step": 430
766
+ },
767
+ {
768
+ "epoch": 0.9421841541755889,
769
+ "grad_norm": 29.379902210497836,
770
+ "learning_rate": 5.08115039419113e-09,
771
+ "logits/chosen": -2.359375,
772
+ "logits/rejected": -2.375,
773
+ "logps/chosen": -131.0,
774
+ "logps/rejected": -140.0,
775
+ "loss": 1.0182,
776
+ "nll_loss": 0.3359375,
777
+ "rewards/accuracies": 0.59375,
778
+ "rewards/chosen": -1.3046875,
779
+ "rewards/margins": 0.09716796875,
780
+ "rewards/rejected": -1.40625,
781
+ "step": 440
782
+ },
783
+ {
784
+ "epoch": 0.9635974304068522,
785
+ "grad_norm": 27.696617271985165,
786
+ "learning_rate": 2.0184767183584474e-09,
787
+ "logits/chosen": -2.390625,
788
+ "logits/rejected": -2.390625,
789
+ "logps/chosen": -126.5,
790
+ "logps/rejected": -136.0,
791
+ "loss": 1.0113,
792
+ "nll_loss": 0.30859375,
793
+ "rewards/accuracies": 0.612500011920929,
794
+ "rewards/chosen": -1.265625,
795
+ "rewards/margins": 0.091796875,
796
+ "rewards/rejected": -1.359375,
797
+ "step": 450
798
+ },
799
+ {
800
+ "epoch": 0.9850107066381156,
801
+ "grad_norm": 31.008731609830598,
802
+ "learning_rate": 3.4261631135654167e-10,
803
+ "logits/chosen": -2.375,
804
+ "logits/rejected": -2.375,
805
+ "logps/chosen": -140.0,
806
+ "logps/rejected": -152.0,
807
+ "loss": 1.0002,
808
+ "nll_loss": 0.337890625,
809
+ "rewards/accuracies": 0.6312500238418579,
810
+ "rewards/chosen": -1.3984375,
811
+ "rewards/margins": 0.11328125,
812
+ "rewards/rejected": -1.515625,
813
+ "step": 460
814
+ },
815
+ {
816
+ "epoch": 1.0,
817
+ "step": 467,
818
+ "total_flos": 0.0,
819
+ "train_loss": 1.0373174434555674,
820
+ "train_runtime": 5792.1601,
821
+ "train_samples_per_second": 10.31,
822
+ "train_steps_per_second": 0.081
823
+ }
824
+ ],
825
+ "logging_steps": 10,
826
+ "max_steps": 467,
827
+ "num_input_tokens_seen": 0,
828
+ "num_train_epochs": 1,
829
+ "save_steps": 100,
830
+ "stateful_callbacks": {
831
+ "TrainerControl": {
832
+ "args": {
833
+ "should_epoch_stop": false,
834
+ "should_evaluate": false,
835
+ "should_log": false,
836
+ "should_save": true,
837
+ "should_training_stop": true
838
+ },
839
+ "attributes": {}
840
+ }
841
+ },
842
+ "total_flos": 0.0,
843
+ "train_batch_size": 8,
844
+ "trial_name": null,
845
+ "trial_params": null
846
+ }