Online_RLHF
[NeurIPS 2025] A PyTorch implementation of the paper "Provably Efficient Online RLHF with One-Pass Reward Modeling". This repository provides a flexible and modular approach to Online Reinforcement Learning from Human Feedback (Online RLHF).
File Explorer
- __init__.py
- active_train_rm_head.py
- active_train_rm_head_hvp.py
- batch_inference.py
- interactive_chat.py
- online_batch_inference.py
- online_train_rm.py
- online_train_rm_head.py
- online_train_rm_head_hvp.py
- online_train_rm_hvp.py
- serve_rm.py
- train_dpo.py
- train_kd.py
- train_kto.py
- train_ppo.py
- train_ppo_online.py
- train_ppo_ray.py
- train_prm.py
- train_rm.py
- train_rm_head.py
- train_rm_head_hvp.py
- train_rm_head_NewtonStep.py
- train_rm_hvp.py
- train_sft.py
- __init__.py
- process_reward_dataset.py
- prompts_dataset.py
- reward_dataset.py
- sft_dataset.py
- unpaired_preference_dataset.py
- utils.py
- __init__.py
- actor.py
- loss.py
- model.py
- ring_attn_utils.py
- utils.py
- __init__.py
- experience_maker.py
- kl_controller.py
- replay_buffer.py
- __init__.py
- launcher.py
- ppo_actor.py
- ppo_critic.py
- utils.py
- vllm_engine.py
- vllm_worker_wrap.py
- __init__.py
- dpo_trainer.py
- kd_trainer.py
- kto_trainer.py
- ppo_trainer.py
- prm_trainer.py
- rm_active_trainer_head.py
- rm_active_trainer_head_hvp.py
- rm_trainer.py
- rm_trainer_head.py
- rm_trainer_head_hvp.py
- rm_trainer_head_NewtonStep.py
- rm_trainer_hvp.py
- sft_trainer.py
- __init__.py
- deepspeed.py
- deepspeed_utils.py
- __init__.py
- collect_results.py
- compare_rewarded_outputs.py
- convert_to_dataset.py
- distributed_sampler.py
- distributed_util.py
- evaluate_iterations.py
- logging_utils.py
- processor.py
- remote_rm_utils.py
- rm_score_selection.py
- utils.py
- __init__.py
- online_active_rlhf_pipeline.py
- online_deployment_mixture2_llama.py
- online_passive_pipeline.sh
- online_passive_pipeline_full.sh
- online_rlhf_pipeline_ultrafeedback_full.py
- online_active_rlhf_pipeline.py
- online_deployment_ultrafeedback_llama.py
- online_passive_pipeline.sh
- online_active_rlhf_pipeline_qwen.py
- online_deployment_ultrafeedback_qwen.py
- online_passive_pipeline_qwen.sh
- .gitignore
- merge_peft.py
- README.md
- requirements.txt
# Use via CDN
jsDelivrjsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.
Command Glossary
Commands referenced in this DOCs, explained below.
pip install
View Details ▼
pip install
Install Python packages.
pip install {{package1 package2 ...}}
Install one or more packages:
pip install {{package1 package2 ...}} {{[-U|--upgrade]}}
Upgrade all specified packages to the latest version, installing any that are not already present:
pip install {{package}}=={{version}}
Install a specific version of a package:
python
View Details ▼
python
Python language interpreter.
python
Start a REPL (interactive shell):
python {{path/to/file.py}}
Execute a specific Python file:
python -i {{path/to/file.py}}
Execute a specific Python file and start a REPL: