TisaleoGPT2Bot1

This model is a fine-tuned version of IAMRS23/TisaleoGPT2Bot1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.4157
• Perplexity: 4328.5317

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 10
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Perplexity
0.5738	1.5385	100	0.4291	4302.6538
0.4753	3.0769	200	0.4127	4327.5306
0.401	4.6154	300	0.4111	4329.6091
0.3544	6.1538	400	0.4127	4321.3208
0.3325	7.6923	500	0.4142	4312.2015
0.3203	9.2308	600	0.4155	4330.6660

Framework versions

• Transformers 4.48.2
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0