pszemraj/mega-small-2048-C1024-tk_id-simplewiki-MR50

mega-small-2048 on simple wikipedia

MEGA for masked LM 'small' (12 layers, 512 hidden size, 2048 ctx in chunks of 1024) on the pszemraj/simple_wikipedia_LM dataset. It achieves the following results on the evaluation set:

• Loss: 3.4773
• Accuracy: 0.4591

Model description

See config for architecture details. While not a ready 'pretrained' model, this was trained from scratch.

This model uses the tokenizer from roberta-base.

Intended uses & limitations

More information needed

Training and evaluation data

Note: this was trained in bf16. the official recommendation is fp32 - still exploring this.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0005
• train_batch_size: 1
• eval_batch_size: 1
• seed: 3208
• gradient_accumulation_steps: 64
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-07
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 3.0

Additionally:

• mask rate of 50% (See paper for details)
• whole-word masking

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
7.2691	0.11	50	7.1000	0.0677
7.1597	0.22	100	6.8388	0.0794
6.5476	0.33	150	6.4004	0.1359
6.5335	0.44	200	6.1776	0.1708
5.7228	0.55	250	5.6106	0.2437
5.4574	0.66	300	5.1391	0.2884
5.2275	0.78	350	4.8626	0.3174
4.9589	0.89	400	4.6454	0.3374
4.6406	1.0	450	4.4498	0.3578
4.8251	1.11	500	4.3055	0.3706
4.4728	1.22	550	4.1877	0.3821
4.3975	1.33	600	4.0709	0.3955
4.4245	1.44	650	3.9909	0.4045
4.2613	1.55	700	3.8976	0.4128
4.1806	1.66	750	3.8515	0.4177
3.9469	1.77	800	3.7883	0.4227
3.9563	1.88	850	3.7314	0.4306
4.0063	1.99	900	3.6975	0.4336
3.9274	2.1	950	3.6561	0.4378
3.788	2.21	1000	3.6280	0.4410
3.8711	2.33	1050	3.5736	0.4467
3.8623	2.44	1100	3.5535	0.4496
3.8575	2.55	1150	3.5407	0.4521
4.0079	2.66	1200	3.5172	0.4543
3.8265	2.77	1250	3.4786	0.4591
3.9513	2.88	1300	3.4741	0.4578
3.554	2.99	1350	3.4773	0.4591

Framework versions

• Transformers 4.33.1
• Pytorch 2.2.0.dev20230907+cu118
• Datasets 2.13.1
• Tokenizers 0.13.3