File size: 2,278 Bytes
01b49dc eb5bfd2 01b49dc eb5bfd2 ee4a8fc 01b49dc eb5bfd2 01b49dc |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 |
---
license: mit
datasets:
- HuggingFaceFW/fineweb-edu
- common-pile/arxiv_papers_filtered
- tiiuae/falcon-refinedweb
- manu/project_gutenberg
- nampdn-ai/tiny-textbooks
- SciPhi/textbooks-are-all-you-need-lite
- abehandlerorg/ccnews
base_model:
- openai-community/gpt2
pipeline_tag: text-generation
---
# GPT-2 from Scratch
This model implements the GPT-2 architecture (125M parameters) trained from scratch.
## Model Description
- **Model type:** GPT-2 (125M parameters)
- **Architecture:** Transformer-based autoregressive language model following the original GPT-2 design
- **Training data:** Uses multiple datasets (check tags) - 18Billion tokens.
- **Language:** English
## Performance and Evaluation
| Dataset | Metric | thecr7guy/gpt2-pretrain | GPT-2 (baseline) |
|----------------|-----------|------------|------------------|
| HellaSwag | acc | **0.291** | 0.289 |
| SciQ | acc | **0.754** | 0.752 |
| Winogrande | acc | 0.491 | **0.516** |
| TruthfulQA MC1 | acc | **0.236** | 0.228 |
| MMLU (overall) | acc | **0.230** | 0.229 |
| - Humanities | acc | 0.242 | 0.242 |
| - Social Sci. | acc | 0.217 | 0.217 |
| - STEM | acc | 0.213 | 0.213 |
| - Other | acc | **0.239** | 0.238 |
## Training Details
- **Training corpus:** Approximately 18B tokens (120GB)
- **Training duration:** 1 epochs (approximately 8 hours total)
- **Hardware:** 8× NVIDIA A100 PCE GPUs via runpod.io
- **Estimated cost:** $ (8*13.52) for complete training
- **Token context:** 1024 tokens
### Hyperparameters
- context_len: 1024
- seed: 42
- epochs: 2
- batch_size: 64
- total_batch_size: 524288 tokens
- grad_clip: 1.0
- optimizer: "adamw"
- max_lr: 6.0e-4
- min_lr: 6.0e-5
- beta1: 0.9
- beta2: 0.95
- weight_decay: 0.1
.
## Commands used during installation
- pip install wandb
- pip install tiktoken
- pip install --upgrade huggingface_hub
- pip install torchinfo
- pip install datasets
- sudo apt update && sudo apt install tmux
- tmux new -s training
- wandb login
- CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 NCCL_P2P_DISABLE=1 \
torchrun --standalone --nproc_per_node=8 train.py
## Contact
GitHub: [thecr7guy2](https://github.com/thecr7guy2) |