C-Optim - GitLite

KO | EN

gitlite — search

#javascript #python #hacktoberfest #typescript #deep-learning #linux #nodejs #java #windows #awesome #cli #kubernetes #vue #awesome-list

C-Optim

(★ 419)

[ICLR 2026] When it comes to optimizers, it's always better to be safe than sorry

File Explorer

Download Latest Version (.zip)

- llama_100m.json
- llama_130m.json
- llama_1b.json
- llama_20m.json
- llama_250m.json
- llama_350m.json
- llama_35m.json
- llama_3b.json
- llama_40m.json
- llama_60m.json
- llama_71m.json
- llama_7b.json
- llama_9m.json
- ppo.py
- ppo_config.py
- ppo_trainer.py
- c_adamw.png
- c_adamw_post_training.png
- c_optim_overview.png
- c_optim_results.png
- c_optim_toy.png
- llama100M_param_sweep.png
- ppo_tldr.png
- - args_utils.cpython-39.pyc
  - dataloader.cpython-39.pyc
  - modeling_llama.cpython-39.pyc
  - training_utils.cpython-39.pyc
- args_utils.py
- dataloader.py
- modeling_llama.py
- training_utils.py
c_adafactor.py
c_adamw.py
c_lion.py
c_muon.py
LICENSE
lion.py
moonlight_toy_train.py
muon.py
nanogpt_speedrun.py
opt_utils.py
post_training.py
ppo_tldr.py
README.md
requirements.txt
run_mae.py
torchrun_main.py

# Installation Guide

1. Get the code

git clone https://github.com/kyleliang919/C-Optim

Downloads the entire project code from GitHub to your computer.

cd C-Optim

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

---

# Cautious Optimizers (C-Optiom): Improving Training with One Line of Code
AdamW has long been the go-to optimizer for transformer pretraining. For years, the research community has been searching for faster and more stable optimizers, with a focus on achieving only positive outcomes. In this work, we introduce a simple, single-line modification in PyTorch for any momentum-based optimizer. This modification, termed **Cautious Optimizer** (e.g., **C-AdamW** and **C-Lion**), opens the door to improved training performance.

Our theoretical findings reveal that this modification preserves Adam’s Hamiltonian function and retains its convergence guarantees under Lyapunov analysis. Additionally, a new family of optimizers emerges from this insight. Among these, we select the simplest for empirical experiments, achieving up to **1.47x speed-up** on **Llama** and **MAE pretraining**.

---

## 🌟 News
- **[2025-08-22]** Asynced batch update version of C-AdamW, tested on FSDP training up to 1B scale
  - [Llama3 1B (C-AdamW)](https://huggingface.co/kz919/llama3_1b_cautious_chinchilla_8232025)
- **[2025-08-14]** Chinchilla Optimal (20x tokens/parameters) on FineWeb-edu with torchtitan
  - [Llama3 1B (AdamW)](https://huggingface.co/kz919/llama3_1b_chinchilla_8132025)
  - [Llama3 1B (C-AdamW)](https://huggingface.co/kz919/llama3_1b_cautious_chinchilla_8142025)
  - [Llama3 3B (AdamW)](https://huggingface.co/kz919/llama3_3b_chinchilla_8142025)
  - [Llama3 3B (C-AdamW)](https://huggingface.co/kz919/llama3_3b_cautious_chinchilla_8152025)
  
- **[2025-08-07]** [Implementing C-AdamW with parallel apply by popular demand](https://github.com/kyleliang919/C-Optim/blob/main/c_adamw.py)     
- **[2025-01-23]** [PPO (Reinforcement Learning)](https://github.com/kyleliang919/C-Optim/blob/main/ppo_tldr.py)
- **[2025-01-14]** [Post Training experiment on Qwen2.5 1.5B Instruct](https://github.com/kyleliang919/C-Optim/blob/main/post_training.py)
- **[2024-12-03]** 🤗🤗🤗 More validation runs on ViTs [timm-optim-caution](https://huggingface.co/rwightman/timm-optim-caution)
- **[2024-12-03]** 🤗🤗🤗 Caution implemented in [huggingface/pytorch-image-models](https://github.com/huggingface/pytorch-image-models/blob/main/timm/optim/adamw.py#L132-L136).
- **[2024-11-24]** Pre-release paper available on arXiv: [Cautious Optimizers: Improving Training with One Line of Code](https://arxiv.org/abs/2411.16085).
- **[2024-11-24]** Official implementation of C-Optim released! Experiment with C-AdamW and C-Lion today.

---

## 🚀 Implementation

### Generic Single-Line Implementation for C-Optim
<div align="center">
  <img src="images/c_optim_overview.png" alt="Image 1" style="width: 550px; margin: 0 auto;">
</div>

---

### Pretraining Results
<div align="center">
  <img src="images/llama100M_param_sweep.png" alt="Image 2" style="width: 650px; margin: 0 auto;">
</div>
<div align="center">
  <img src="images/c_optim_results.png" alt="Image 3" style="width: 650px; margin: 0 auto;">
</div>

---

### Post Training Results
<div align="center">
  <img src="images/c_adamw_post_training.png" alt="Image 4" style="width: 650px; margin: 0 auto;">
</div>

---

### PPO
<div align="center">
  <img src="images/ppo_tldr.png" alt="Image 5" style="width: 650px; margin: 0 auto;">
</div>

---

## 📦 Installation
### Install Experiment Dependencies
```bash
pip install -r requirements.txt
```

---

## 🛠️ Usage
### Pretraining Llama on C4
```bash
torchrun --standalone --nproc_per_node 1 torchrun_main.py \
         --model_config configs/llama_60m.json \
         --lr 0.001 \
         --batch_size 16 \
         --total_batch_size 512 \
         --activation_checkpointing \
         --num_training_steps 10000 \
         --warmup_steps 1000 \
         --weight_decay 0 \
         --grad_clipping 1.0 \
         --dtype bfloat16 \
         --eval_every 1000 \
         --single_gpu \
         --optimizer c-adamw \
         --max_length 1024
```

### Pretraining MAE on ImageNet 1K (50 Epochs)
```bash
torchrun --standalone --nproc_per_node 4 run_mae.py \
    --dataset_name ILSVRC/imagenet-1k \
    --output_dir ./vit-mae-c \
    --remove_unused_columns False \
    --label_names pixel_values \
    --mask_ratio 0.75 \
    --norm_pix_loss \
    --do_train \
    --do_eval \
    --base_learning_rate 1.5e-4 \
    --lr_scheduler_type cosine \
    --weight_decay 0.05 \
    --num_train_epochs 50 \
    --warmup_ratio 0.05 \
    --per_device_train_batch_size 256 \
    --per_device_eval_batch_size 8 \
    --logging_strategy steps \
    --logging_steps 10 \
    --eval_strategy epoch \
    --save_strategy epoch \
    --load_best_model_at_end True \
    --save_total_limit 3 \
    --seed 1337 \
    --custom_optim c-adamw \
    --trust_remote_code \
    --gradient_accumulation_steps 4
```
### Post Training Qwen2.5
```
torchrun \
    --rdzv_id=$JOB_ID \
    --rdzv-backend=c10d \
    --nnodes=1:8 \
    --nproc-per-node=1 \
    --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
    post_training.py --model "Qwen/Qwen2.5-1.5B-Instruct" \
                     --output_dir cautious_1.5b \
                     --per_device_train_batch_size 1 \
                     --gradient_accumulation_steps 2 \
                     --max_length 8192 \
                     --cautious
```

---

### PPO
```
accelerate launch ppo_tldr.py \
    --dataset_name trl-internal-testing/tldr-preference-sft-trl-style \
    --dataset_test_split validation \     
    --output_dir models/minimal/ppo_tldr \
    --learning_rate 3e-6 \
    --per_device_train_batch_size 16 \
    --gradient_accumulation_steps 4 \
    --total_episodes 1000000 \
    --model_name_or_path EleutherAI/pythia-1b-deduped \
    --sft_model_path cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr \
    --reward_model_path cleanrl/EleutherAI_pythia-1b-deduped__reward__tldr \
    --local_rollout_forward_batch_size 16 \
    --missing_eos_penalty 1.0 \
    --stop_token eos \
    --eval_strategy steps \
    --eval_steps 100 \
    --custom_optim c_adamw \
    --num_gpus 8
```

---

## 📖 Citation
```bibtex
@article{liang2024cautious,
  title={Cautious optimizers: Improving training with one line of code},
  author={Liang, Kaizhao and Chen, Lizhang and Liu, Bo and Liu, Qiang},
  journal={arXiv preprint arXiv:2411.16085},
  year={2024}
}
```

---