rewardedsoups - GitLite

KO | EN

gitlite — search

#python #react #llm #machine-learning #rust #linux #docker #nodejs #windows #awesome #cli #kubernetes #awesome-list #security

rewardedsoups

(★ 65)

Rewarded soups official implementation

파일 탐색기

최종 버전 다운로드 (.zip)

- - __init__.py
  - coco_dataloader.py
  - coco_dataset.py
  - transparent_data_loader.py
- - __init__.py
  - eval.py
  - get_stanford_models.sh
- - __init__.py
  - loss.py
  - reward.py
- - __init__.py
  - captioning_model.py
  - End_ExpansionNet_v2.py
  - ensemble_captioning_model.py
  - ExpansionNet_v2.py
  - layers.py
  - swin_transformer_mod.py
- - radam.py
- - data_generator.py
  - test.py
  - train.py
- - args_utils.py
  - language_utils.py
  - masking.py
  - saving_utils.py
- README.md
- - assistant.py
  - review.py
  - stack.py
  - summary.py
- - args_utils.py
  - inference_utils.py
  - llama_utils.py
  - ppo_utils.py
- inference_rewardedsoups.py
- README.md
- requirements.txt
- train_ppo.py
- - styles.css
- - 01_✍️_News_summarization.py
  - 02_🎨_Image_generation.py
  - 03_🦿_Locomotion.py
- home.py
- requirements.txt
- utils.py
.gitignore
__init__.py
fig_intro.png
README.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/alexrame/rewardedsoups

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd rewardedsoups

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r llama/requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. NeurIPS 2023.

[Alexandre Ramé](https://alexrame.github.io/), [Guillaume Couairon](https://scholar.google.com/citations?user=O1DeDyEAAAAJ&hl), [Mustafa Shukor](https://scholar.google.com/citations?user=lhp9mRgAAAAJ), [Corentin Dancette](https://cdancette.fr/), [Jean-Baptiste Gaya](https://www.isir.upmc.fr/personnel/gaya), [Laure Soulier](https://www.isir.upmc.fr/personnel/soulier/) and [Matthieu Cord](http://webia.lip6.fr/~cord/)

Repository to reproduce the main results from our [paper](https://arxiv.org/abs/2306.04488). The interactive demos are available on our [website](https://alexrame-rewardedsoups-streamlit-apphome-pxh21g.streamlit.app/) or our [HuggingFace space](https://huggingface.co/spaces/alexrame/rewardedsoups). The reviews are available on [openreview](https://openreview.net/forum?id=lSbbC2VyCu&noteId=np0m5ACZtx).

![](https://raw.githubusercontent.com/alexrame/rewardedsoups/main/fig_intro.png)

## Abstract

Foundation models are first pre-trained on vast unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably from human feedback (RLHF), can further align the network with the intended usage. Yet the imperfections in the proxy reward may hinder the training and lead to suboptimal results; the diversity of objectives in real-world tasks and human opinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following a multi-policy strategy. Rather than focusing on a single a priori reward, we aim for Pareto-optimal generalization across the entire space of preferences. To this end, we propose rewarded soup, first specializing multiple networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we show that the weights remain linearly connected when fine-tuned on diverse rewards from a shared pre-trained initialization. We demonstrate the effectiveness of our approach for text-to-text (summarization, Q&A, helpful assistant, review), text-image (image captioning, text-to-image generation, visual grounding, VQA), and control (locomotion) tasks. We hope to enhance the alignment of deep models, and how they interact with the world in all its diversity.

## Experiments

We have implemented rewarded soups across a variety of standard learning tasks: text-to-text generation, image captioning, image generation, visual grounding, and locomotion. We follow a systematic procedure:

- we independently optimize diverse rewards on training samples. For all tasks, we employ the default architecture, hyperparameters and RL algorithm; the only variation being the reward used across runs.
- we evaluate the rewards on the test samples: the results are visually represented in series of plots in the paper.
- we verify the linear mode connectivity by examining whether RS's rewards exceed the interpolated rewards.
- we compare the front defined by RS when sliding $\lambda$ between $0$ and $1$ to the MORL's solutions optimizing the $\mu$-weighted rewards.

When possible, we have released the shareable implementations under the corresponding folders. More experiments will be released. The procedures are further detailed in the corresponding `README.md`.

### Text-to-text experiments: LLaMA with diverse RLHFs

The code in [llama](llama/README.md) enables the reproduction of our main experiments, where we fine-tune LLaMA-7b, instruction fine-tuned on Alpaca, on five tasks: summarization (on news or on reddit), answering of stack exchange questions, movie review generation or conversational assistant.

### Image-to-text experiments: captioning with diverse statistical rewards

The code in [captioning](captioning/README.md) enables the reproduction of our image-to-text experiments, where we fine-tune the state-of-the-art weights from ExpansionNet v2 on various statistical metrics.

## Citation

If you find this code useful for your research, please consider citing our work:

```
@inproceedings{rame2023rewardedsoups,
  title   = {Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards},
  author  = {Ram{\'e}, Alexandre and Couairon, Guillaume and Shukor, Mustafa and Dancette, Corentin and Gaya, Jean-Baptiste and Soulier, Laure and Cord, Matthieu},
  year    = {2023},
  booktitle = {NeurIPS},
}
```