pre-training-for-dummies

KO | EN

gitlite — search

#javascript #python #react #golang #machine-learning #rust #deep-learning #docker #java #kubernetes #awesome-list #pytorch #ios #openai

pre-training-for-dummies

(★ 13)

Train GPT-2 from scratch on a single GPU. Companion code for the Pre-Training from Scratch book.

pre-training-for-dummies Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Pre-Training for Dummies

Train GPT-2 (124M parameters) from scratch on a single GPU. No abstractions, no hand-waving.

This is the companion code for the book *Pre-Training from Scratch* by Elliot Arledge.

## Requirements

- Python 3.10+
- PyTorch 2.0+ with CUDA
- GPU with 24GB VRAM (RTX 3090, RTX 4090, A100, etc.)

```bash
pip install torch numpy tiktoken
pip install datasets tqdm  # only needed for OpenWebText
```

## Quick Start (Shakespeare)

Test the setup with Shakespeare (~1M tokens, trains in minutes):

```bash
# 1. Prepare data
python prepare_data.py --dataset shakespeare

# 2. Train (reduced config for quick test)
python train.py --max_iters 5000 --eval_interval 500 --log_interval 100

# 3. Generate
python generate.py --prompt "To be or not to be"
```

## Full Training (OpenWebText)

Train GPT-2 124M on OpenWebText (~9B tokens):

```bash
# 1. Prepare data (downloads ~50GB, takes a few hours)
python prepare_data.py --dataset openwebtext

# 2. Train (takes several days on a single GPU)
python train.py --compile

# 3. Generate
python generate.py --prompt "The meaning of life is"
```

## Training Details

Default configuration matches GPT-2 124M:

| Parameter | Value |
|-----------|-------|
| Layers | 12 |
| Heads | 12 |
| Embedding dim | 768 |
| Context length | 1024 |
| Vocab size | 50304 |
| Parameters | 124M |

Training hyperparameters:

| Parameter | Value |
|-----------|-------|
| Batch size | 12 |
| Gradient accumulation | 40 steps |
| Effective batch | 491K tokens |
| Learning rate | 6e-4 (peak) |
| LR schedule | Cosine decay |
| Warmup | 2000 steps |
| Weight decay | 0.1 |
| Total iterations | 600K |

## Files

- `model.py` - GPT-2 model implementation
- `train.py` - Training loop with mixed precision, gradient accumulation
- `prepare_data.py` - Data preparation for Shakespeare and OpenWebText
- `generate.py` - Text generation from trained checkpoints

## Expected Results

After full training on OpenWebText:

- Train loss: ~3.0
- Val loss: ~3.1
- Perplexity: ~22

The model will generate coherent paragraphs but may drift or repeat. This is a base model, not fine-tuned for chat.

## Resume Training

```bash
python train.py --resume out/ckpt.pt
```

## Acknowledgments

Based on [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy.

## License

MIT

pre-training-for-dummies

pre-training-for-dummies Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

pre-training-for-dummies

pre-training-for-dummies Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README