pods

KO | EN

gitlite — search

#agents #ai-agents #llm #kotlin #deep-learning #framework #docker #golang #animation #tensorflow

(★ 11)

No description available.

File Explorer

ZIP Download

- test.yaml
- train.yaml
- - a100s_length_curve.pdf
  - a100s_length_curve.png
  - a100s_main.pdf
  - a100s_main.png
- - length_over_time.pdf
  - length_over_time.png
  - main.pdf
  - main.png
- - length_over_time.pdf
  - length_over_time.png
  - main.pdf
  - main.png
- - length_over_time.pdf
  - length_over_time.png
  - main.pdf
  - main.png
- - length_over_time.pdf
  - length_over_time.png
  - main.pdf
  - main.png
- - h100s_length_curve.pdf
  - h100s_length_curve.png
  - h100s_main.pdf
  - h100s_main.png
- - length_over_time.pdf
  - length_over_time.png
  - main.pdf
  - main.png
- - length_over_time.pdf
  - length_over_time.png
  - main.pdf
  - main.png
- - plan-of-attack.png
- - .gitkeep
- - README.md
- - benchmark_e2b.py
  - decontaminate.py
  - e2b_router.py
  - generate_reasoning.py
  - get_tensor_parallel_size.py
  - morph_router.py
  - run_benchmarks.py
  - upload_details.py
- - e2b_router.slurm
  - evaluate.slurm
  - generate.slurm
  - morph_router.slurm
  - README.md
  - serve_r1.slurm
  - serve_router.slurm
  - train.slurm
- - __init__.py
  - test_rewards.py
- exp.sh
- LICENSE
- Makefile
- README.md
- setup.cfg
- setup.py
- a100s-acc.csv
- a100s-length.csv
- base-llama.json
- base-qwen.json
- h100s-acc.csv
- h100s-length.csv
- Instruct-GSM8K-512-128-16-Run1.json
- Instruct-GSM8K-512-128-16-Run10.json
- Instruct-GSM8K-512-128-16-Run2.json
- Instruct-GSM8K-512-128-16-Run3.json
- Instruct-GSM8K-512-128-16-Run4.json
- Instruct-GSM8K-512-128-16-Run5.json
- Instruct-GSM8K-512-128-16-Run6.json
- Instruct-GSM8K-512-128-16-Run7.json
- Instruct-GSM8K-512-128-16-Run8.json
- Instruct-GSM8K-512-128-16-Run9.json
- Instruct-GSM8K-512-16-16-Run1.json
- Instruct-GSM8K-512-16-16-Run10.json
- Instruct-GSM8K-512-16-16-Run2.json
- Instruct-GSM8K-512-16-16-Run3.json
- Instruct-GSM8K-512-16-16-Run4.json
- Instruct-GSM8K-512-16-16-Run5.json
- Instruct-GSM8K-512-16-16-Run6.json
- Instruct-GSM8K-512-16-16-Run7.json
- Instruct-GSM8K-512-16-16-Run8.json
- Instruct-GSM8K-512-16-16-Run9.json
- Instruct-GSM8K-512-256-16-Run1.json
- Instruct-GSM8K-512-256-16-Run10.json
- Instruct-GSM8K-512-256-16-Run2.json
- Instruct-GSM8K-512-256-16-Run3.json
- Instruct-GSM8K-512-256-16-Run4.json
- Instruct-GSM8K-512-256-16-Run5.json
- Instruct-GSM8K-512-256-16-Run6.json
- Instruct-GSM8K-512-256-16-Run7.json
- Instruct-GSM8K-512-256-16-Run8.json
- Instruct-GSM8K-512-256-16-Run9.json
- Instruct-GSM8K-512-32-16-Run1.json
- Instruct-GSM8K-512-32-16-Run10.json
- Instruct-GSM8K-512-32-16-Run2.json
- Instruct-GSM8K-512-32-16-Run3.json
- Instruct-GSM8K-512-32-16-Run4.json
- Instruct-GSM8K-512-32-16-Run5.json
- Instruct-GSM8K-512-32-16-Run6.json
- Instruct-GSM8K-512-32-16-Run7.json
- Instruct-GSM8K-512-32-16-Run8.json
- Instruct-GSM8K-512-32-16-Run9.json
- Instruct-GSM8K-512-64-16-Run1.json
- Instruct-GSM8K-512-64-16-Run10.json
- Instruct-GSM8K-512-64-16-Run2.json
- Instruct-GSM8K-512-64-16-Run3.json
- Instruct-GSM8K-512-64-16-Run4.json
- Instruct-GSM8K-512-64-16-Run5.json
- Instruct-GSM8K-512-64-16-Run6.json
- Instruct-GSM8K-512-64-16-Run7.json
- Instruct-GSM8K-512-64-16-Run8.json
- Instruct-GSM8K-512-64-16-Run9.json
- Instruct-GSM8K-512-64-2-Run1.json
- Instruct-GSM8K-512-64-2-Run10.json
- Instruct-GSM8K-512-64-2-Run2.json
- Instruct-GSM8K-512-64-2-Run3.json
- Instruct-GSM8K-512-64-2-Run4.json
- Instruct-GSM8K-512-64-2-Run5.json
- Instruct-GSM8K-512-64-2-Run6.json
- Instruct-GSM8K-512-64-2-Run7.json
- Instruct-GSM8K-512-64-2-Run8.json
- Instruct-GSM8K-512-64-2-Run9.json
- Instruct-GSM8K-512-64-4-Run1.json
- Instruct-GSM8K-512-64-4-Run10.json
- Instruct-GSM8K-512-64-4-Run2.json
- Instruct-GSM8K-512-64-4-Run3.json
- Instruct-GSM8K-512-64-4-Run4.json
- Instruct-GSM8K-512-64-4-Run5.json
- Instruct-GSM8K-512-64-4-Run6.json
- Instruct-GSM8K-512-64-4-Run7.json
- Instruct-GSM8K-512-64-4-Run8.json
- Instruct-GSM8K-512-64-4-Run9.json
- Instruct-GSM8K-512-64-8-Run1.json
- Instruct-GSM8K-512-64-8-Run10.json
- Instruct-GSM8K-512-64-8-Run2.json
- Instruct-GSM8K-512-64-8-Run3.json
- Instruct-GSM8K-512-64-8-Run4.json
- Instruct-GSM8K-512-64-8-Run5.json
- Instruct-GSM8K-512-64-8-Run6.json
- Instruct-GSM8K-512-64-8-Run7.json
- Instruct-GSM8K-512-64-8-Run8.json
- Instruct-GSM8K-512-64-8-Run9.json
- Instruct-GSM8K-MaxR-512-64-16-Run1.json
- Instruct-GSM8K-MaxR-512-64-16-Run10.json
- Instruct-GSM8K-MaxR-512-64-16-Run2.json
- Instruct-GSM8K-MaxR-512-64-16-Run3.json
- Instruct-GSM8K-MaxR-512-64-16-Run4.json
- Instruct-GSM8K-MaxR-512-64-16-Run5.json
- Instruct-GSM8K-MaxR-512-64-16-Run6.json
- Instruct-GSM8K-MaxR-512-64-16-Run7.json
- Instruct-GSM8K-MaxR-512-64-16-Run8.json
- Instruct-GSM8K-MaxR-512-64-16-Run9.json
- Instruct-GSM8K-Rand-512-64-16-Run1.json
- Instruct-GSM8K-Rand-512-64-16-Run10.json
- Instruct-GSM8K-Rand-512-64-16-Run2.json
- Instruct-GSM8K-Rand-512-64-16-Run3.json
- Instruct-GSM8K-Rand-512-64-16-Run4.json
- Instruct-GSM8K-Rand-512-64-16-Run5.json
- Instruct-GSM8K-Rand-512-64-16-Run6.json
- Instruct-GSM8K-Rand-512-64-16-Run7.json
- Instruct-GSM8K-Rand-512-64-16-Run8.json
- Instruct-GSM8K-Rand-512-64-16-Run9.json
- Instruct-Llama-512-16-16-0.04-Run1.json
- Instruct-Llama-512-16-16-0.04-Run10.json
- Instruct-Llama-512-16-16-0.04-Run2.json
- Instruct-Llama-512-16-16-0.04-Run3.json
- Instruct-Llama-512-16-16-0.04-Run4.json
- Instruct-Llama-512-16-16-0.04-Run5.json
- Instruct-Llama-512-16-16-0.04-Run6.json
- Instruct-Llama-512-16-16-0.04-Run7.json
- Instruct-Llama-512-16-16-0.04-Run8.json
- Instruct-Llama-512-16-16-0.04-Run9.json
- Instruct-Llama-512-64-16-0.04-Run1.json
- Instruct-Llama-512-64-16-0.04-Run10.json
- Instruct-Llama-512-64-16-0.04-Run2.json
- Instruct-Llama-512-64-16-0.04-Run3.json
- Instruct-Llama-512-64-16-0.04-Run4.json
- Instruct-Llama-512-64-16-0.04-Run5.json
- Instruct-Llama-512-64-16-0.04-Run6.json
- Instruct-Llama-512-64-16-0.04-Run7.json
- Instruct-Llama-512-64-16-0.04-Run8.json
- Instruct-Llama-512-64-16-0.04-Run9.json
- Instruct-MATH-NF-512-32-8-Run1.json
- Instruct-MATH-NF-512-32-8-Run10.json
- Instruct-MATH-NF-512-32-8-Run2.json
- Instruct-MATH-NF-512-32-8-Run3.json
- Instruct-MATH-NF-512-32-8-Run4.json
- Instruct-MATH-NF-512-32-8-Run5.json
- Instruct-MATH-NF-512-32-8-Run6.json
- Instruct-MATH-NF-512-32-8-Run7.json
- Instruct-MATH-NF-512-32-8-Run8.json
- Instruct-MATH-NF-512-32-8-Run9.json
- Instruct-MATH-NF-512-8-8-Run1.json
- Instruct-MATH-NF-512-8-8-Run10.json
- Instruct-MATH-NF-512-8-8-Run2.json
- Instruct-MATH-NF-512-8-8-Run3.json
- Instruct-MATH-NF-512-8-8-Run4.json
- Instruct-MATH-NF-512-8-8-Run5.json
- Instruct-MATH-NF-512-8-8-Run6.json
- Instruct-MATH-NF-512-8-8-Run7.json
- Instruct-MATH-NF-512-8-8-Run8.json
- Instruct-MATH-NF-512-8-8-Run9.json
- length.py
- merge.py
- plot-a100s.py
- plot-h100s.py
- plot.py
- __init__.py
- maxrewardgrpo_config.py
- maxrewardgrpo_trainer.py
- maxvargrpo_config.py
- maxvargrpo_trainer.py
- randomgrpo_config.py
- randomgrpo_trainer.py
- data.py
- eval.py
- grader.py
- normalize.py
- unsloth.py
.gitignore
.python-version
evaluate-base.py
evaluate-run.py
LICENSE
pyproject.toml
README.md
train.py
uv.lock

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Table of Contents

- [General Information](#general-information)
- [Reproducing the Experiments](#reproducing-the-experiments)
- [License](#license)
- [Citation](#citation)
- [Contact](#contact)

## General Information

This repository contains the source code for the experiments in paper ["Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning"](https://arxiv.org/abs/2504.13818) by Yixuan Even Xu*, Yash Savani*, Fei Fang, Zico Kolter. We implemented GRPO-PODS (Policy Optimization with Down-Sampling) and compared its performance with vanilla GRPO. Our implementation is based on [Unsloth](https://github.com/unslothai/unsloth) and [OpenR1](https://github.com/huggingface/open-r1).

## Reproducing the Experiments

- To install relevant dependencies, install `uv` and enter

``` bash
  uv sync
  ```

- To re-run the single-GPU experiments, edit `config/train.yaml`, and enter

``` bash
  mkdir -p checkpoints
  uv run python3 train.py
  ```

- To evaluate the saved checkpoints of a single-GPU experiment run, edit `config/test.yaml`, and enter

``` bash
  uv run python3 evaluate-run.py
  ```

- To run the multi-GPU experiments, `cd` into the `open-r1` directory, follow the install instructions in the `README.md` file within the directory, and then run the following script.

``` bash
  bash exp.sh
  ```

The data can be collected and downloaded from the corresponding wandb runs and plotted using the plotting scripts.

- To generate the plots in the paper, enter

``` bash
  uv run python3 scripts/plot.py
  uv run python3 scripts/plot-h100s.py
  uv run python3 scripts/plot-a100s.py
  ```
  
## License

This repository's source code is available under the [Apache-2.0 License](LICENSE).

## Citation

If you use this code in your research, please cite our paper:

```bibtex
@article{xu2025not,
  title={Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning},
  author={Xu, Yixuan Even and Savani, Yash and Fang, Fei and Kolter, Zico},
  journal={arXiv preprint arXiv:2504.13818},
  year={2025}
}
```

## Contact

For any questions or issues, please contact us via email:
- Yixuan Even Xu: yixuanx@cs.cmu.edu
- Yash Savani: ysavani@cs.cmu.edu

pods

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

pods

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README