opsd-predictive-law

KO | EN

gitlite — search

#kubernetes #awesome-list #cross-platform #macos #anthropic #go #javascript #developer-tools #css #cloud-native

opsd-predictive-law

(★ 9)

A Predictive Law for On-Policy Self-Distillation From World Feedback (ICML RLxF 2026)

File Explorer

ZIP Download

- - code.py
  - gpqa.py
  - math.py
  - mmlu_pro.py
  - prompts.py
  - sciknoweval.py
  - train.py
  - utils.py
- - code.py
  - code_elo.py
  - codeforces.py
  - data_handling.py
  - humaneval.py
  - livecodebench.py
  - math.py
  - mbpp.py
  - primeintellect.py
  - taco.py
- load_dataset.py
- prepare_lcb.sh
- preprocess.py
- split_tests.py
- - run_eval_self_teacher_gap_1step.sh
  - run_sdpo_lcb.sh
- fig1_predictive_law.png
- fig2_convergence.png
- fig3_scaling.png
- verl_training.sh
- - __init__.py
  - base.py
  - nccl_checkpoint_engine.py
  - nixl_checkpoint_engine.py
  - README.md
- - __init__.py
- - __init__.py
  - base.py
  - gsm8k_interaction.py
  - weather_interaction.py
- - __init__.py
  - __main__.py
  - base_model_merger.py
  - fsdp_model_merger.py
  - megatron_model_merger.py
- - __init__.py
  - README.md
  - registry.py
  - weight_loader_registry.py
- - __init__.py
- - __init__.py
- - __init__.py
  - base_tool.py
  - geo3k_tool.py
  - gsm8k_tool.py
  - image_zoom_in_tool.py
  - mcp_base_tool.py
  - mcp_search_tool.py
  - sandbox_fusion_tools.py
  - schemas.py
  - search_tool.py
- - __init__.py
  - constants_ppo.py
  - fsdp_sft_trainer.py
  - main_eval.py
  - main_generation.py
  - main_generation_server.py
  - main_ppo.py
  - runtime_env.yaml
  - sft_trainer.py
  - sft_trainer_ray.py
- - __init__.py
  - activation_offload.py
  - attention_utils.py
  - chat_template.py
  - config.py
  - device.py
  - distributed.py
  - flops_counter.py
  - fs.py
  - fsdp_utils.py
  - groupwise.py
  - hdfs_io.py
  - import_utils.py
  - logging_utils.py
  - megatron_peft_utils.py
  - megatron_utils.py
  - memory_buffer.py
  - memory_utils.py
  - model.py
  - net_utils.py
  - npu_flash_attn_utils.py
  - py_functional.py
  - ray_utils.py
  - rollout_skip.py
  - rollout_trace.py
  - seqlen_balancing.py
  - tensordict_utils.py
  - tokenizer.py
  - torch_dtypes.py
  - torch_functional.py
  - tracking.py
  - transferqueue_utils.py
  - transformers_compat.py
  - ulysses.py
- - version
- - __init__.py
  - engine_workers.py
  - fsdp_workers.py
  - megatron_workers.py
- __init__.py
- base_config.py
- protocol.py
- py.typed
.envrc
.gitignore
.python-version
eval_self_teacher_gap_1step.sbatch
LICENSE
pyproject.toml
README.md
run.sbatch
uv.lock

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Paper: https://arxiv.org/abs/2605.30070

## Summary

On-policy self-distillation (OPSD) trains a student policy to match a *self-teacher* - the same model conditioned on privileged context (e.g. environment feedback, peer solutions, expert hints). The choice of privileged context drives OPSD's effectiveness, but its impact on final performance has been hard to anticipate without running full training.

We show that the **initial student–self-teacher accuracy gap**, measured before any training, is a strong **linear predictor** of the **final student improvement** after OPSD post-training. The relationship holds across:

- Six qualitatively different privileged-context constructions (none, expert preamble, environment feedback, LLM peer hints, own solution + feedback, peer solution + feedback)
- Two model families (Qwen3, Olmo 3)
- Four model sizes (Qwen3 0.6B / 1.7B / 4B / 8B)

The practical consequence is that you can screen OPSD configurations cheaply before launching a full training run and predict gains at larger scales from cheaper measurements.

## Main result

![Predictive law](https://raw.githubusercontent.com/Tufalabs/opsd-predictive-law/master/figures/fig1_predictive_law.png)

For both Qwen3-8B and Olmo-3-7B-Instruct, the initial student–self-teacher gap linearly predicts final student improvement on LiveCodeBench v6 (Δ mean@4). OLS fits: Qwen3-8B R² = 0.949, Olmo-3-7B R² = 0.996. Each point is one privileged-context configuration averaged over 3 seeds.

## Training dynamics

![Convergence](https://raw.githubusercontent.com/Tufalabs/opsd-predictive-law/master/figures/fig2_convergence.png)

During 50 steps of OPSD on Qwen3-8B with the Peer Solution + Feedback context, the student's mean@4 (purple) steadily converges toward the self-teacher's (blue). The self-teacher also gains slightly over training, raising the ceiling.

## Scaling

![Scaling](https://raw.githubusercontent.com/Tufalabs/opsd-predictive-law/master/figures/fig3_scaling.png)

The same linear law holds across Qwen3-{0.6B, 1.7B, 4B, 8B} with the Peer Solution + Feedback context fixed (OLS R² = 0.977), suggesting the gap is a scale-invariant predictor of OPSD improvement.

## Install

Environment is managed with [`uv`](https://docs.astral.sh/uv/). Python 3.12.

```bash
uv sync                        # creates .venv from pyproject.toml + uv.lock
source .venv/bin/activate
echo "WANDB_API_KEY=..." > .env  # sourced by run.sbatch
```

Direnv users can `cd` in and the `.envrc` does the venv activation automatically.

## Run

Default: 50-step OPSD on Qwen3-8B with the `all` (Peer Solution + Feedback) context.

```bash
sbatch run.sbatch
```

Knobs in `run.sbatch`:

- `CONSTRUCTION_METHOD` (line 52) — one of `all`, `feedback_only`, `correct_solution_processed`, `self_correct_solution_with_feedback`, `none`, `static_teacher1`, etc.
- The model arg on the trailing `bash ./experiments/rich_feedback/run_sdpo_lcb.sh ...` line

LiveCodeBench v6 train/test parquets are produced automatically by `data/prepare_lcb.sh` on first run (downloads from HuggingFace).

## Tested on

| Item | Value |
|---|---|
| Hardware (paper experiments) | NVIDIA B200 GPUs (≈720 B200-hours total) |
| Cluster (this repo) | SLURM, 4 GPUs / run |
| OS | Linux x86\_64 |
| Python | 3.12 |
| CUDA toolkit | 12.8 (PyTorch cu128 wheels) |
| PyTorch | 2.9.1 |
| vLLM | 0.15.1 |
| flash-attn | 2.8.3 |
| Container image | `docker://the41/cuda-dev` |

## Attribution

Built on top of [SDPO](https://github.com/lasgroup/SDPO) (Hübotter et al., 2026) and [verl](https://github.com/verl-project/verl) (Sheng et al., 2025).

opsd-predictive-law

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

opsd-predictive-law

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README